Cassandra : Replaced Node Shows In Nodetool GossipInfo And Nodetool Status

Question

We are using Cassandra 3.9.0. Recently we had some trouble regarding 1 node. This node was crashed as 100% disk usage was reached.

One approach we are thinking to replace the node by a new node, according to following instruction provided by Datastax. https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsReplaceNode.html

After the replacement is done in a Test Env, when we do nodetool status from new node, the old node does not show up. But, when executed from other nodes, the old dead node shows up. Similarly, when nodetool gossipinfo is executed in existing nodes other than new incoming node, reference of the old node is found.

As shown below, we are replacing a2 by a4

Status=Up/Down
/ State=Normal/Leaving/Joining/Moving
--  Address     Load  Tokens  Owns(effective)  Host ID  Rack
UN  x.x.x.a1  4.52 GiB   256      72.0%       HOSTID1  rack1
DN  x.x.x.a2  4.56 GiB   256      77.5%       null     rack1
UN  x.x.x.a3  4.33 GiB   256      76.9%       HOSTID3  rack1
UN  x.x.x.a4  5.59 GiB   256      73.6%       HOSTID4  rack1

When node tool status is run from new incoming node which is the replacement node, we get results as below.

UN  x.x.x.a1  4.52 GiB   256      100.0%    HOSTID1  rack1
UN  x.x.x.a3  4.33 GiB   256      100.0%    HOSTID3  rack1
UN  x.x.x.a4  5.59 GiB   256      100.0%    HOSTID4  rack1

Is there any recommended way to solve this situation?

Answer 1

That doc page outlines a process slightly different from the one I use to replace nodes, and doesn't seem to mention running either nodetool decommission or nodetool removenode. I don't want to make any assumptions about your cluster (for instance, you might be multi-DC), but I believe you will have to run one of those commands in order to have the cluster remove the dead node from the topology.

Since it sounds like you already terminated the instance that the "dead node" was running on, you won't be able to run nodetool decommission from it. Instead, I'd recommend going on another node, one that still sees it as part of the cluster, and running nodetool removenode. The command takes the UUID of the dead node as an argument, so you can find that via nodetool status to pass in.

The command is long-running so I recommend running it in a screen or tmux session. You can check the progress by running nodetool removenode -- status. The command will redistribute tokens that the dead node held ownership over to the other nodes in the cluster.

EDIT Just want to clarify that what I meant by the process outlined in the doc you posted being different from my own, I was referring specifically to running the new node with the -Dcassandra.replace_address=address_of_dead_node option. In any case, If the node is dead, and unable to rejoin the cluster, there's no harm in running nodetool removenode on its UUID.

Answer 2

The other option you should have try that replace the same node if all good on that node. I meant own node replacement in this case no range movement for data and it will have same range and will stream the data from the other nodes according to the token range.