We're using the Elixir Xandra library and recently, the change was made to use the Cassandra async requests and moving away from the DBConnection library, which was basically synchronous.
While investigating a memory leak, I have noticed that some requests for some stream ids never get a response. The throughput we are inserting at is 1000 rows/second, accross 3 nodes, so around 333 writes/s per node. We have 10 connections to each node which makes around 33 requests per connection per second to a node.
While inserting synchronously, we never got timeouts in the same test environment. However now, we get around 30 timeouts in 5 minutes per connection, meaning 900 timeouts in 5 minutes with this rate. I wanted to ask whether this is a known issue and whether we could do something about this in the Xandra client to prevent this happening? Is there maybe some limitations around the interval of 2 requests coming from the same connection?
We're using the Open Source Cassandra 4.0.10