OpenProcess call returns pseudo handle

Question

MSDN says that OpenProcess() must return either a valid handle, or NULL on error.

However, I met a rare situation on Win7 x64 (and also on Win 8.1 x86, Win XP x64, Win Vista x64) where OpenProcess() returned -1 for the current process, i.e. the pseudo handle, while I enumerated processes. It occurs very rarely from time to time (when I run my test suite on different platforms). And I can't reproduce it on Win 10.

Then CloseHandle() fails on this handle with ERROR_INVALID_HANDLE error. But on the other hand MSDN says that

The pseudo handle need not be closed when it is no longer needed. Calling the CloseHandle function with a pseudo handle has no effect.

Why does this happen? Is it correct behavior for OpenProcess()?

Below is an example of my code:

HANDLE snap = CreateToolhelp32Snapshot (TH32CS_SNAPPROCESS, 0);
PROCESSENTRY32 prEntry = {};
prEntry.dwSize = sizeof (PROCESSENTRY32);
if (Process32First (snap, &prEntry))
{
   do
   {
       if (prEntry.th32ProcessID)
       {
           HANDLE h = OpenProcess(PROCESS_QUERY_INFORMATION, FALSE, prEntry.th32ProcessID);
           // <<<< from time to time h is (-1) here when meets the current process
           if (h)
           {
               wchar_t imageFilename[MAX_PATH + 1] = {};
               if (GetProcessImageFileName(hProc, imageFilename, MAX_PATH))
               {
                   // do something
               }
               if (!CloseHandle(h))
               {
                   // CLoseHandle returns FALSE on pseudo handle (at least on Win 7)
                   DWORD err = ::GetLastError();
                   // err == 0x6, i.e. ERROR_INVALID_HANDLE, for the pseudo handle
                   // << create a memory dump here for further analysis
               }
           }
       }
   }
   while (Process32Next (snap, &prEntry));
}

UPD: I found why my tests never fail on Win 10 platform - CloseHandle(HANDLE(-1)) always returns TRUE on Win 10, while on Win 7 it returns FALSE with 0x6 error. But I still have no explanation about OpenProcess() behaviour.

SOLUTION: As Ben Voigt said (see the accepted answer) there was a hook in the test environment, which was not cleared after the previous test suite run. After days of debugging the hook was found and localized, the test suite was fixed to clean the hooks up. Now OpenProcess call works correctly.

Answer 1

The most likely explanation is a poorly written OpenProcess hook, for example both antimalware and malware use such hooks.

Writer of this hook didn't read the OpenProcess documentation carefully, and when real Windows OpenProcess succeeds but the hook wants to block your access, it is doing

return INVALID_HANDLE_VALUE;

(This could be a copy/paste error from also writing a CreateFile hook)

Due to the fact such hooks are predominantly used in rootkits -- both "evil" and "good" (but accidentally evil anyway) varieties, you probably won't be able to confirm the hook from inside the system. A kernel debugger connection, on the other hand, should show that during real Windows OpenProcess execution, the return address is to the hook code, not the kernel transition stub.