cqlsh connection shows below error for couple of machine, but it is successful in other machines in cluster. Bad nodes were removed out of cluster, system_auth replication is 5 for both dc's with NetworkTopologyStrategy. Anything I'm missing to check here ?
Connection error: ('Unable to connect to any servers', {'0.0.0.0': AuthenticationFailed('Failed to authenticate to 0.0.0.0: Error from server: code=0100 [Bad credentials] message="Unable to perform authentication: Operation timed out - received only 0 responses."',)})
./cqlsh --debug --ssl
Using CQL driver: <module 'cassandra' from '/usr/local/akamai/dart_apache_cassandra/bin/../lib/cassandra-driver-internal-only-3.11.0-bb96859b.zip/cassandra-driver-3.11.0-bb96859b/cassandra/__init__.py'>
Using connect timeout: 5 seconds
Using 'utf-8' encoding
Using ssl: True
Connection error: ('Unable to connect to any servers', {'0.0.0.0': AuthenticationFailed('Failed to authenticate to 0.0.0.0: Error from server: code=0100 [Bad credentials] message="Unable to perform authentication: Operation timed out - received only 0 responses."',)})
1.These nodes consistently produce this error when attempting to connect via cqlsh? 2. Output of nodetool status look normal 3. System log show below errors, all nodes are in UN status, describecluster show all nodes are in one schema version
WARN [OptionalTasks:1] 2024-02-15 14:48:59,872 CassandraRoleManager.java:379 - CassandraRoleManager skipped default role setup: some nodes were not ready
INFO [OptionalTasks:1] 2024-02-15 14:48:59,872 CassandraRoleManager.java:418 - Setup task failed with error, rescheduling