We experience a very rare crash on our single MongoDb instance. System was good for 'years', running in a VMWare VM with 16Gb of memory. Now in short time we experience twice the:
WT_CURSOR.search: read checksum error for 4096B block at offset 20169220096: block header checksum of 538976288 does not match expected checksum of 2817788143.
Found the location where the error is generated in wired tiger code (block_read.c). Figured out that the checksum reported (538976288) is actually 0x20202020 which is 4 times space character.
Too nice a number to be random. Load is ever growing on that system, we are not good at throwing old data away. Error we can 'solve' by throwing away the collection with error but that is after crash and with data loss.
Any pointers where to look, my suspicion is a write out of bounds but no proof so far :-(