opengl32.dll hangs on SetPixelFormat How to replace or hook a custom function

Question

I have a scenario where in some cases when opengl32.setPixelFormat is called it then call _wglDescribePixelFormat ( which is exported function ) which then calls further in the call stack ComputeBitsFromMasks and finally _MaskToBitsAndShift. The _MaskToBitsAndShift function remains in the loop and never finishes, causing hanging the whole app. Callstack is

opengl32.dll!_MaskToBitsAndShift@12
opengl32.dll!ComputeBitsFromMasks
opengl32.dll!___wglGetBitfieldColorFormat@16
opengl32.dll!_InternalDescribePixelFormat@40
opengl32.dll!_wglDescribePixelFormat@16
opengl32.dll!_wglSetPixelFormat@12
gdi32full.dll!_SetPixelFormat@12
Unmanaged.dll!CGLContext::CreateOffScreenContext
Unmanaged.dll!Shared::Generate3DImages

Decoding the _MaskToBitsAndShift method from asm to c becomes like this.

Asm code

opengl32!MaskToBitsAndShift:
6da7d6f2 8bff            mov     edi,edi
6da7d6f4 55              push    ebp
6da7d6f5 8bec            mov     ebp,esp
6da7d6f7 56              push    esi
6da7d6f8 8b7508          mov     esi,dword ptr [ebp+8]
6da7d6fb 33c0            xor     eax,eax
6da7d6fd 40              inc     eax – initialize eax to 1, used to find highest set bit in the mask
6da7d6fe c60200          mov     byte ptr [edx],0
6da7d701 c60600          mov     byte ptr [esi],0
6da7d704 84c8            test    al,cl
6da7d706 7508            jne     opengl32!MaskToBitsAndShift+0x1e (6da7d710)
6da7d708 03c0            add     eax,eax – shifts eax (the bit) left 
6da7d70a fe06            inc     byte ptr [esi] – increases shift count
6da7d70c 85c1            test    ecx,eax – ecx is the mask, first param to this function apparently
– and this is the problem. if ecx or mask is 0, test ecx,eax will never set the flag for je to fail, hence infinite loop
6da7d70e 74f8            je      opengl32!MaskToBitsAndShift+0x16 (6da7d708)
6da7d710 5e              pop     esi
6da7d711 eb04            jmp     opengl32!MaskToBitsAndShift+0x25 (6da7d717)
6da7d713 03c0            add     eax,eax
6da7d715 fe02            inc     byte ptr [edx]
6da7d717 85c1            test    ecx,eax
6da7d719 75f8            jne     opengl32!MaskToBitsAndShift+0x21 (6da7d713)
6da7d71b 5d              pop     ebp
6da7d71c c20400          ret     4

Equivalant c++ code

void stdcall MaskToBitsAndShift(DWORD mask, BYTE* shiftCount, BYTE* bitCount)
{
// literal translation from asm; could be safer using bittest intrinsics or counted bits but no big deal
DWORD bit = 1;
*shiftCount = 0;
*bitCount = 0;

while (!(mask & bit))

{ *shiftCount += 1; bit <<= 1; }
while (mask & bit)

{ *bitCount += 1; bit <<= 1; }
}

If we notice the mask if 0, the loop will never end. I cannot hook this function because it is not exported.

Any idea about working around it ? or rewriting the _wglDescribePixelFormat method which is exported but I have no idea how to tranlate that.

The initial call to the setPixelFormat is

bool CGLContext::CreateOffScreenContext(int nWidth, int nHeight, CString* pstrLog /*=NULL*/)
{
    if(pstrLog)
        pstrLog->Append("Creating Off Screen Context\r\n");

    m_eContextType = CONTEXT_TYPE_OFFSCREEN;

    // Create a new Device Context
    m_pDC = new CDC;
    m_pDC->CreateCompatibleDC(NULL);

    // Initialize all of the compenents used to select a pixel format
    memset(&m_PixelFormatDescriptor, 0, sizeof(m_PixelFormatDescriptor));
    m_PixelFormatDescriptor.nSize = sizeof(PIXELFORMATDESCRIPTOR);
    m_PixelFormatDescriptor.nVersion = 1;
    m_PixelFormatDescriptor.dwFlags = PFD_DRAW_TO_BITMAP|PFD_SUPPORT_OPENGL|PFD_SUPPORT_GDI;
    m_PixelFormatDescriptor.iPixelType = PFD_TYPE_RGBA;
    m_PixelFormatDescriptor.cColorBits = 0;
    m_PixelFormatDescriptor.cDepthBits = 0;
    m_PixelFormatDescriptor.cAccumBits = 0;
    m_PixelFormatDescriptor.cStencilBits = 0;
    m_PixelFormatDescriptor.cAuxBuffers = 0;
    m_PixelFormatDescriptor.iLayerType = PFD_MAIN_PLANE;

    m_hDC = m_pDC->m_hDC;
    
    int nPixelFormat = ChoosePixelFormat(m_hDC, &m_PixelFormatDescriptor);

    if(nPixelFormat == 0) {
        if(pstrLog)
        {
            CString str;
            str.Format("Unable to Choose Pixel Format: %li\r\n", nPixelFormat);
            pstrLog->Append(str);
            pstrLog->Append(DescribePFD(m_PixelFormatDescriptor));
            return false;
        }
        else
        {   
            ThrowException("Unable to Choose Pixel Format");
        }
    }.....