Cassandra crashes with Out Of Memory within minutes after starting

Question

We have a Cassandra cluster with 3 nodes and replication factor 3 on AWS using EC2Snitch.

Instance type is c5.2xlarge (8 core and 16GB RAM).

The cluster had been working fine but suddenly since yesterday evening, the cassandra process on all the nodes started crashing. They are set to restart automatically but then they crash with Out of Memory Heap Space error in 1 or 2 or 3 minutes after start.

Heap configs:

MAX_HEAP_SIZE="4G"
HEAP_NEWSIZE="800M"

After this, we tried increasing the node size to r5.4x or 128 GB memory and assigned 64GB Heap but still the same thing happens, irrespective of all 3 nodes being started or only one node being started at a time. We could note that first garbage collection happens after some time and then consecutively within seconds, failing to free any further memory and eventually crashing.

We are not sure what is being pulled to memory immediately after starting.

Other parameters:

• Cassandra version : 2.2.13
• Database size is 250GB
• hinted_handoff_enabled: true
• commitlog_segment_size_in_mb: 64
• memtable_allocation_type: offheap_buffers

Any help here, would be appreciated.

Edit: We found that there is particular table when queried, it causes the casssandra node to crash.

cqlsh:my_keyspace> select count(*) from my_table ;
ReadTimeout: Error from server: code=1200 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

So we think, it is related to the data being corrupt/huge in this particular table. Thanks.

Answer 1

Some quick observations:

• If you're building a new cluster, use the latest 3.11.x version. There's no point in building new on 2.2.
• Based on your settings, it looks like you're using CMS GC. If you're not overly familiar with GC tuning, you may get more stability by switching to G1, and not specifying a HEAP_NEWSIZE (G1 figures out Eden sizing on its own).
• If you're stuck on CMS, the guidance for setting HEAP_NEWSIZE at 100mb x cores, is wrong. To avoid new->old gen promotion, set HEAP_NEWSIZE at 40%-50% of total heap size and increase MaxTenuringThreshold to something like 6-8.
• On a 16GB RAM machine with CMS GC, I would use a 8GB heap, and flip memtable_allocation_type: offheap_buffers back to heap_buffers.
• Set commitlog_segment_size_in_mb back to 32. Usually when folks need to mess with that, it's to lower it, unless you've also changed max_mutation_size_in_kb.
• You haven't mentioned what the application is doing when the crash happens. I suspect that a write-heavy load is happening. In that case, you may need more than 3 nodes, or look at rate-limiting the number of in-flight writes on the application side.

Additional info to help you:

CASSANDRA-8150 - A Cassandra committer discussion on good JVM settings.

Amy's Cassandra 2.1 Tuning Guide - Amy Tobey's admin guide has a lot of wisdom on good default settings for cluster configuration.

Edit

We are using G1 GC.

It is very, VERY important that you not set a heap new size (Xmn) with G1. Make sure that gets commented out.

select count(*) from my_table ;

Yes, unbound queries (queries without WHERE clauses`) will absolutely put undue stress on a node. Especially if the table is huge. These types of queries are something that Cassandra just doesn't do well. Find a way around using/needing this result.

You might be able to engineer this to work by setting your paging size smaller (driver side), or by using something like Spark. Or maybe with querying by token range, and totaling the result on the app-side. But you'll be much better off not doing it.

Answer 2

In addition to the CG and memory tuning suggestions by @aaron, you should also check that you are using the right compaction strategy for your data.

https://docs.datastax.com/en/dse/5.1/dse-dev/datastax_enterprise/config/configChooseCompactStrategy.html#Whichcompactionstrategyisbest

You should also check for corrupt SStables, as trying to fetch corrupted data will also manifest in the same way. (for example https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/tools/toolsScrub.html)