Every few months on a Windows server where I have a few .NET Core 3.1 web applications running in IIS, I see TCP port exhaustion errors being logged, with all the associated symptoms happening (unable to open more network ports to connect to database server, domain controller, etc.)
Looking at netstat -anob traces when this happens, the number of active connections is surprisingly only in the hundreds range for the whole system, and there's only a handful in TIME_WAIT state.
However, looking at the number of handles held by processes with Process Explorer and handle.exe, it turns out one website (w3wp process) has 31000 handles and growing, most of which are \Device\Afd file handles, which seems to point to a network handle leak problem.
Restarting the website in IIS solves the issue.
- How come the huge number of handles doesn't show at all in the TCP ports open in
netstattraces? You would think a port exhaustion problem is visible in a tool that is supposed to report network stats. - More generally, how do
\Device\Afdhandles relate to TCP ports in the system/network stack? Troubleshoot pages like this one mention them but don't explain a lot. - Any idea what type of problem manifests itself like that? Suggestions for tools to to identify the faulty part of the website/app?