I have installed Cassandra 2.0 On CentOS6.5 Server and and while testing simple records everything is working fine, Now I have to upload 600 billion rows, when I use COPY on cqlsh it failed after 5 minutes and approx rows inserted are 0.2 million with rpc timeout, then I opted for pycasso and parsed csv and tried to import using inserts commands, after every 10K records, We opted to close connection and develop new connection again. but after around 60k records it failed with with timeout.
My debug trace shows something this while server is not accepting inserts, without any activity it's still busy.
DEBUG [OptionalTasks:1] 2014-05-30 04:34:16,305 MeteredFlusher.java (line 41) Currently flushing 269227480 bytes of 2047868928 max
DEBUG [OptionalTasks:1] 2014-05-30 04:34:17,306 MeteredFlusher.java (line 41) Currently flushing 269227480 bytes of 2047868928 max
DEBUG [OptionalTasks:1] 2014-05-30 04:34:18,306 MeteredFlusher.java (line 41) Currently flushing 269227480 bytes of 2047868928 max
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy for schema_triggers is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy for compaction_history is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy for batchlog is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy for sstable_activity is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy for peer_events is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy for compactions_in_progress is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy for hints is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy for schema_keyspaces is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy for range_xfers is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy for schema_columnfamilies is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy for NodeIdInfo is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy for paxos is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy for schema_columns is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,014 ColumnFamilyStore.java (line 298) retryPolicy for IndexInfo is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,014 ColumnFamilyStore.java (line 298) retryPolicy for peers is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,014 ColumnFamilyStore.java (line 298) retryPolicy for local is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,307 MeteredFlusher.java (line 41) Currently flushing 269227480 bytes of 2047868928 max
DEBUG [OptionalTasks:1] 2014-05-30 04:34:20,307 MeteredFlusher.java (line 41) Currently flushing 269227480 bytes of 2047868928 max
DEBUG [OptionalTasks:1] 2014-05-30 04:34:20,716 ColumnFamilyStore.java (line 298) retryPolicy for backup_calls is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:20,716 ColumnFamilyStore.java (line 298) retryPolicy for sessions is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:20,716 ColumnFamilyStore.java (line 298) retryPolicy for events is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:21,308 MeteredFlusher.java (line 41) Currently flushing 269
while When I try to insert records it show error like this in debug log.
DEBUG [OptionalTasks:1] 2014-05-30 04:34:40,717 ColumnFamilyStore.java (line 298) retryPolicy for backup_calls is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:40,717 ColumnFamilyStore.java (line 298) retryPolicy for sessions is 0.99
DEBUG [OptionalTasks:1] 2014-05-30 04:34:40,718 ColumnFamilyStore.java (line 298) retryPolicy for events is 0.99
DEBUG [Thrift:24] 2014-05-30 04:34:40,775 CustomTThreadPoolServer.java (line 211) Thrift transport error occurred during processing of message.
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
DEBUG [Thrift:19] 2014-05-30 04:34:40,775 CustomTThreadPoolServer.java (line 211) Thrift transport error occurred during processing of message.
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
DEBUG [Thrift:21] 2014-05-30 04:34:40,775 CustomTThreadPoolServer.java (line 211) Thrift transport error occurred during processing of message.
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
DEBUG [Thrift:1] 2014-05-30 04:34:40,775 CustomTThreadPoolServer.java (line 211) Thrift transport error occurred during processing of message.
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
DEBUG [Thrift:21] 2014-05-30 04:34:40,789 CassandraServer.java (line 949) batch_mutate
DEBUG [Thrift:19] 2014-05-30 04:34:41,314 CassandraServer.java (line 949) batch_mutate
DEBUG [OptionalTasks:1] 2014-05-30 04:34:41,317 MeteredFlusher.java (line 41) Currently flushing 269227480 bytes of 2047868928 max
DEBUG [Thrift:24] 2014-05-30 04:34:41,858 CassandraServer.java (line 949) batch_mutate
Any Idea about what is happening?, after insertion failed it will not work until server is restarted.
Node Details:
There is only a single node and details are given below,
# nodetool describecluster
Cluster Information:
Name: Test Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
93403140-5bd6-3cfa-9973-b18df44227f9: [127.0.0.1]
My Node is up and running?
# nodetool status
Note: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns Host ID Token Rack
UN 127.0.0.1 35.05 MB 100.0% 6bc340b2-4534-400f-b23d-9fe3ed0c6971 -9120530846114649324 rack1
Server Details:
OS CentOS release 6.5 (Final)
hdd 8Tb all SSD Drives
RAM 32GB
Proccessor: model name : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz, 24 Core(2 physical
processors with 12 cores each)