This is what OS X's built in color inversion feature can turn your screen into:

It can invert all colors, turn them grayscale, adjust contrast. Now I want to build my own implementation and therefore need a professionals' advice.
Inability to capture inverted screen made me thinking that inversion is a sort of an adjustment layer, that resides above all windows and simply is not exposed to the interaction events. Is that so? Is it done via OpenGL libs?
I don't look into actual coding help, but rather a design/approach on solving the problem. In my goal app I will need to define output color diapasons and apply color transformation rules (input => output).
Thanks in advance.

The way Mac OS does the color inversion is (probably) by using Quartz Display Services to modify the graphics card's gamma table.
Graphics cards have two of these tables to modify color output after composition into the final frame buffer. One of them can be modified by applications to alter the way the screen shows any RGB value.
Here's code to invert the display: