How to differentiate between socket handle and file handle

Question

I need to check certain behavior in process creation events in windows, i need to implement a rule that check the startupinfo structure passed to createprocess api call and extracting the std input/std output handles values for the created process. Then i have to check if this handle is belong to tcp socket or not. Is there any api function that might help me to get any info about the handle number i have (whether it's file handle or socket handle)?

Answer 1

Use the GetFileType() function

Retrieves the file type of the specified file.

Syntax

DWORD WINAPI GetFileType( _In_ HANDLE hFile ); 

Parameters

hFile [in]

A handle to the file.

Return value

The function returns one of the following values.

FILE_TYPE_CHAR

The specified file is a character file, typically an LPT device or a console.

FILE_TYPE_DISK

The specified file is a disk file.

FILE_TYPE_PIPE

The specified file is a socket, a named pipe, or an anonymous pipe.

FILE_TYPE_REMOTE

Unused.

FILE_TYPE_UNKNOWN

Either the type of the specified file is unknown, or the function failed

Answer 2

Building on @RemyLebeau's answer, I thought I would see if I could find a reliable way to distinguish a socket from a pipe (which GetFileType() cannot do), and I came up with the following, which seems to work and has no obvious downside.

The gist of it is that getsockopt() will return WSAENOTSOCK (= 10038) if passed anything that is not a SOCKET. Therefore, this test is sufficient in and of itself, just so long as any handle you pass to it is either a SOCKET or a file or pipe HANDLE. Don't just pass it any old HANDLE (there are all sorts of these), or it might get confused as per @HansPassant's first comment below.

Sample code:

#include <winsock2.h>                   // * before* windows.h (!)
#include <windows.h>
#include <assert.h>
#include <iostream>

int main ()
{
    WSADATA wsa_data;
    int err = WSAStartup (2, &wsa_data);
    assert (err == 0);

    // Pipe

    HANDLE hReadPipe, hWritePipe;
    BOOL ok = CreatePipe (&hReadPipe, &hWritePipe, NULL, 2048);
    assert (ok);

    int opt;
    int optlen = sizeof (opt);
    err = getsockopt ((SOCKET) hReadPipe, IPPROTO_TCP, TCP_NODELAY, (char *) &opt, &optlen);

    if (err)
    {
        DWORD dwErr = GetLastError ();
        std::cout << "Pipe: " << dwErr << std::endl;
    }
    else
        std::cout << "Pipe: OK" << std::endl;

    CloseHandle (hReadPipe);
    CloseHandle (hWritePipe);

    // Socket

    SOCKET skt = WSASocket (AF_INET, SOCK_STREAM, 0, NULL, 0, 0);
    assert (skt != INVALID_SOCKET);

    optlen = sizeof (opt);
    err = getsockopt (skt, IPPROTO_TCP, TCP_NODELAY, (char *) &opt, &optlen);

    if (err)
    {
        DWORD dwErr = GetLastError ();
        std::cout << "Socket: " << dwErr << std::endl;
    }
    else
        std::cout << "Socket: OK" << std::endl;

    closesocket (skt);
    return 0;
}

Output:

Pipe: 10038
Socket: OK

---

Edit: If you read the comments below you will see there has been some discussion about whether this code can be led to believe that a file or pipe HANDLE is actually a SOCKET. Well, it can't. We can know this because functions like ReadFile() and WriteFile() work equally well on both file / pipe HANDLE's and SOCKET's, and if there were any chance of mistaking one for the other then that would not work.

So, this code is (a) safe, (b) simple, and (c) effective in all situations (including redirected output, which Remy's code will think is a socket). I therefore recommend it. Just make sure you call WSAStartup() before you do anything else.

Thank you to @HansPassant and @eryksun for making significant contributions to this post.

Answer 3

Use GetNamedPipeInfo(s, NULL, NULL, NULL, NULL) to differentiate pipe from socket.

bool is_socket(LPVOID s)
{
  if (GetFileType(s) != FILE_TYPE_PIPE) return false;
  return !GetNamedPipeInfo(s, NULL, NULL, NULL, NULL);
}