In reverse-engineering an old music format I've encountered a conversion algorithm that converts a 14-bit unsigned floating-point value to an 11-bit integer value. The algorithm is as follows (pvf is the 14-bit floating-point value):
i87 = ((pvf >> 8) * 8) // 7
m = (((i87 & 0x7) + 7) << 8) | (pvf & 0xff)
e = (~i87 & 0x38) >> 3
p = m >> e
After playing with it some I've discovered that for the relevant values [0,56*256) this is equivalent to:
m = (pvf % 0x700) + 0x700 # Mantissa with hidden bit
e = 7 - (pvf // 0x700)
p = m >> e
This algorithm is... mysterious. It's obviously a floating-point to integer conversion, shifting a mantissa by an exponent. Where things get strange is the encoding of the high byte. The basic format is 00pppppp mmmmmmmm. When p is multiplied by 8/7 (and m is left alone) this produces 00eeennn mmmmmmmm with n being the high bits of the mantissa. In other words, rather than e getting 3 bits and n getting 3, n gets log2(7) = ~2.8 bits (n can be 0-6 but not 7) and e is mixed in with it. But WHY? On the architecture this is implemented on this is DRASTICALLY slower than simply allocating a full 3 bits to e and n, so there must be some good reason for it.
The only clue is exactly what this format is used for. Note frequencies are stored in this format, and the resulting p is the period value written to hardware where
Hz = 1789773 // (16 * (p + 1))
I note that the algorithm does NOT subtract 1 from p, which I suspect to be a bug. This appears to be done to piecewise-approximate linear pitch slides/vibrato, as relative pitch (an integer) is added to the floating-point value BEFORE conversion. But how does that relate to the 8/7 multiplier?
As examples:
- 0x375c: 0x37 * 8/7 = 0x3e: 0x3e5c (111110 01011100) with e = 7 - 7 = 0 and n = 6 + 7 = 0xd (the hidden bit). p = 0xd5c >> 0 = 0xd5c (32.70 Hz ~= C1).
- 0x14: 0x14 (00010100) with e = 7 - 0 = 7 and n = 0 + 7 = 7. p = 0x714 >> 7 = 0xe (7457 Hz ~= A#8; actually supposed to be B8, but the p off by one bug is causing severe detuning at high frequencies).