I have a challenge for you all that all of our other reverse enginnering communities's absolute best efforts have been unable to complete or make progress on. We're part of groups of teams who are attempting to decompile and match entire source codes of retro video games, but there's one function in particular that is just flooring us, and it's only about 20-30 lines of C. We have a collaboration website called decomp.me used to share "scratches" of these attempts to make it easy for anyone to help out in just one click, but I'm not here to advertise that, this is about that one function thats been a problem for us for the last 2 years or so.
Here is the challenge: https://decomp.me/scratch/Td7XY
If you are curious, the source code for that website is here.
I presume you all have a question though; why are we resorting to Stack Overflow for this particular function?
This is an unmatched C function circa written around 1995-1996, compiled for MIPS (specifically MIPS II), written in C (specifically C89/C90). We are absolutely stumped as literally every other related function in the same file(s) have been matched already with sane code and exhaustively proven compiler+settings. This function belongs to a set of functions as part of a crash screen debugger file written for the Nintendo 64. How do we know its part of the same file, since linker-wise could split them at any point, you say? On the Nintendo 64, assembler objects are padded to multiple of 0x10s (hexadecimal respectively). If the last function of a file stops before a multiple of 0x10, the MIPS assembler adds nops until it does. With multiple games as a reference for this system its quite easy to know with absolute certainty this is part of the same "crash_screen.c" file.
Ok; how do we know the compiler, which is IDO 5.3? (IRIS Development Option 5.3 for IRIX 5.3, circa 1994) There were multiple releases of IDO all targeting MIPS II and III and respectively.
For the Nintendo 64's target CPU architecture, there was actually a problem with the hardware; early retail unit CPU revisions of the r4300 have an errata with the mul instruction; if two muls are in a row (specifically the main slot and delay slot both are occupied I believe) the 2nd mul's result can be incorrect. To fix this, SGI released a custom assembler patch which just adds a nop between offending instructions, delaying the pipeline until the result is correct. We believe this is because the 2nd mul returns a result before it actually finishes, so delaying the pipeline with a nop is the correct workaround here. This patch however was for IDO 5.3 only, and built into IDO 7.1, which was not out until late November 1996, too late to have compiled this function. Not sure if I will specify which game this is from here, but because of the release window and the time it takes to manufacture the cartridges we can safely rule out 7.1. Plus, some of the loop codegen changes with 7.1 and most of our ROM doesnt match anymore. IDO 5.3 has been thoroughly proven as the compiler.
From there, it wasnt all that hard to deduce the settings. It's all standard stuff; -O2 and the like. No surprises.
Ok, now we're at the actual function itself. I wanted to get that out of the way since I would rather this question not be filled with doubts and inquiries about the compiler itself. Those inquiries are off topic for this post. So why ask Stack Overflow for help? Well this function is doing something very strange and thats loading the value 6 into 5 different registers. We have not been able to reproduce this except with UB and I am only mostly sure that's a red herring and not the real solution. I believe it has something to do with loop unrolling which caused a value calculated in a loop (as 6) to get duplicated 6 times. IDO is actually fairly decent at de-duplicating code like this, but unfortunately for some reason it failed with this function.
All the function is doing is drawing a 6x6 square to a frame buffer (a 16-bit array respectively). But whatever it's doing, its probably written very ameteurish-ly, in my opinion. This is also from the earliest known revision of this system. Part of the issue here is that the output loads the number 6 into 5 different registers for some reason, but we think it has to do with the loop being unrolled. Im really more so asking for people here to just try writing the code in that scratch in a bunch of ways and see if the score drops a lot. The lower the score is better.
Can anyone here solve what we could not? How was this function written so that when compiled it matches? When matching, it will display a purple checkmark on the right side of the decomp.me scratch.
We've tried while loops, do while loops, for loops, and some forms of UB. One UB that actually did cause the duplication is if we had one of the variables be uninitialized and then later added that to a variable like 6 + uninitialized. This would cause the duplication, but break the other part of the function, so it seemed like a red herring. That UB scratch can be seen here https://decomp.me/scratch/VqeQw as an example.
I'm mainly looking for just how many different ways this function can be written. Obscure C syntax, intern code styles and choices, all of those. I'm looking for as much info on any possible permutations for this function.
The square looks like this and starts at 40px wh.
The code draws the red square first, then the white square. The outter-most loop runs twice.
Also, row/column are labelled backwards from what you would expect.
row places the cursor on the next line and column moves the cursor left to right (the inner-most loop).
Note that white-space matters with this compiler. Placing code on the same line can on occasion create differing code-gen.
m2c output (custom n64 mips to C decompiler) https://decomp.me/scratch/kaKZe (this is where decomping the func began)
ida output: https://pastebin.com/Uyw3mDPt