We do frequent bulk deletion + reloading of data in RocksDB. To ensure that the deleted data does not keep occupying disk space, I set the periodic_compaction_seconds to 1 day. This causes RocksDB to reclaim some disk space, but however the overall disk space used continues to grow gradually unless I do a full compaction by calling CompactRange(rocksdb::CompactRangeOptions(), nullptr, nullptr).
Can someone explain to me why the periodic compaction is not doing a good job of purging deleted data from disk?
Here's what the compaction stats look like before and after the CompactRange() call:
BEFORE
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 0/0 0.00 KB 0.0 19.2 0.0 19.2 28.4 9.2 0.0 3.1 34.0 50.3 577.53 57.75 3147 0.184 15M 12K 0.0 0.0
L1 12/0 416.42 MB 0.8 30.2 9.2 21.0 28.1 7.1 0.0 3.1 75.9 70.5 407.60 69.19 25 16.304 41M 186K 0.0 0.0
L2 157/0 3.33 GB 0.7 9.8 7.0 2.8 4.2 1.4 0.0 0.6 24.3 10.3 415.53 181.80 32 12.985 20M 529K 0.0 0.0
L3 721/0 38.24 GB 0.8 1.8 1.8 0.0 1.8 1.8 0.0 1.0 21.5 21.5 87.76 84.15 1 87.762 11M 24 0.0 0.0
L4 120/0 7.52 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
Sum 1010/0 49.50 GB 0.0 61.0 18.1 43.0 62.5 19.5 0.0 6.8 42.0 43.0 1488.42 392.89 3205 0.464 89M 728K 0.0 0.0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
AFTER
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 1/0 1.04 KB 0.5 19.2 0.0 19.2 28.4 9.2 0.0 3.1 34.0 50.3 577.54 57.75 3149 0.183 15M 12K 0.0 0.0
L1 0/0 0.00 KB 0.0 30.2 9.2 21.0 28.1 7.1 0.0 3.1 75.9 70.5 407.60 69.19 26 15.677 41M 187K 0.0 0.0
L2 0/0 0.00 KB 0.0 13.6 7.5 6.1 6.5 0.3 0.0 0.9 25.8 12.3 537.90 297.89 35 15.369 39M 9389K 0.0 0.0
L3 0/0 0.00 KB 0.0 42.5 4.1 38.4 10.2 -28.3 0.0 2.5 47.7 11.4 913.29 884.19 9 101.477 221M 165M 0.0 0.0
L4 130/0 8.03 GB 0.0 13.9 6.3 7.6 6.3 -1.3 1.8 1.0 35.4 16.0 403.91 389.24 2 201.953 79M 40M 0.0 0.0
Sum 131/0 8.03 GB 0.0 119.4 27.1 92.3 79.4 -13.0 1.8 8.6 43.1 28.6 2840.24 1698.25 3221 0.882 397M 215M 0.0 0.0
Int 0/0 0.00 KB 0.0 23.9 6.8 17.2 8.2 -9.0 1.8 8235570.7 40.8 13.9 600.84 579.93 6 100.140 125M 82M 0.0 0.0
I'm using the default RocksDB configuration values.
The
periodic_compaction_secondsconfiguration only compacts files to the same level. This is a bit unexpected and there is an attempt to fix this behavior: https://github.com/facebook/rocksdb/issues/12165