I'm trying to reproduce a deleted data reappearance issue with a 2 node cluster, RF=2. I did with follow steps:
- alter table gc_grace_second to 180s.
- count table's rows, 8 rows
- shutdown node2
- delete 4 rows at node1
- wait 180s then compact the table
- restart node2 and count table's rows, 8 rows; count tables row at node1: 4 rows.
- nodetool repair -full
- check rows count at each node, both have 4 rows.
So after step 8, I'm confused since the official doc says:

So my expectation was that the deleted data will appear after node2 is restarted. But it's disappear after 'repair -full' obviously.
BUT just one hour later, when I recheck the table: count() result is 8 at node1 and 0 at node2. Then I repair -full table again, and 2 nodes have identical count() result: 8. So deleted data just reappeared.
My question is:
- Can this test considered as a successful reproduction of deleted data reappearance?
- Why deleted data didn't reappear immediately right after node2 restart. I mean yeah, it's 8 rows after restart but it changed to 4 rows after repair full. My expectation was node1 count() increasing to 8 rows rather than node1 count() decline to 4 rows.
- Why data reappear one hour ago at last?