Spring app returns CassandraConnectionFailureException due to NoNodeAvailableException: No node was available to execute the query

Question

We have added 1 more dc to existing cluster. Application was deployed in each region separately. After adding the DC & region wise application deployment, We started seeing the below exception even though all nodes in the both data-centers up and running normal. We're using correct data center name in application deployment. I'm not sure why we are seeing these exceptions. The application was running as expected in new datacenter & having issues with existing datacenter and it's intermittent.

DB version: 4.0.1 Driver Version : 4.11.3 DataStax Java OSS driver.

Error Details: org.springframework.data.cassandra.CassandraConnectionFailureException: ReactivePreparedStatementCallback; CQL [INSERT INTO product_by_Itemid (Itemid,data) VALUES (?,?) USING TTL 60]; No node was available to execute the query; nested exception is com.datastax.oss.driver.api.core.NoNodeAvailableException: No node was available to execute the query

Can anyone please suggest what could be the wrong ?

Regards, Mani

Answer 1

The Cassandra Java driver throws NoNodeAvailableException in the following scenarios:

• all nodes are down and unavailable, or
• all the configured contact points are invalid.

Check that you have the correct contact points configured. Do not add hosts or IPs that belong to another DC. Cheers!

Answer 2

If the problem is intermittent as you stated in your description, then I would look at load.

Are you using LOCAL consistency levels?

• If not, that is probably the issue because you have more nodes now, which would require more time and resources. You could look at latency between DC's.
• If so, then I would look at performance-related issues. Are the disks saturated, CPU looks good? How about memory? Also, take a look at the logs, any errors reported there?

Verify that your replication factors are correct as well. And if you changed them, did you repair the data?

Additionally, I would upgrade since 4.0.1 was an early version of 4.x.