On Windows Server 2022 I'm running a custom HTTPS server (part of a larger app). The service binds, listens and then accepts on port 81 for a while and netstat -abn shows:
TCP 0.0.0.0:81 0.0.0.0:0 LISTENING
[MySvc.exe]
All is well. Then after some time the process needs to close so instead of calling accept() again, it instead calls closesocket(). I'm 100% sure it is calling closesocket() on the listening socket (stepped through the debugger, and looked at logs showing socket value).
Not every single time, but frequently I'll see this with netstat:
TCP 0.0.0.0:81 0.0.0.0:0 LISTENING
[System]
The socket was closed and the process is gone, but the System process is "listening" on my port now!
Now the problem. When the service starts again, I end up with:
TCP 0.0.0.0:81 0.0.0.0:0 LISTENING
[MySvc.exe]
TCP 0.0.0.0:81 0.0.0.0:0 LISTENING
[System]
This happens because I set SO_REUSEADDR. If I don't then my service can't bind to the port at all. The problem is now when my service calls accept(), it never gets any connections as long as System is listening. All return codes are checked to bind(), listen() and accept(). If accept is called on a blocking socket, it doesn't return even when clients are attempting to connect.
After maybe 10-30 minutes System stops listening on the port and everything starts working (while my application is running and waiting for accept() to starting working, which it does after the 10-30 minutes).
Things I've checked:
- peer sockets (returned from accept) are closed as much as possible (they are given to OpenSSL, but still tracked and closed). When the process stops there are usually some in TIME_WAIT or maybe CLOSE_WAIT, so I think they were all closed 1.a. making sure to call SSL_shutdown and SSL_free, but also tried without it
- tried just terminating the process with TerminateProcess() with no closesocket() on anything
- made sure WSACleanup is called the same number of times as WSAStartup, also more times, and also not at all
- made sure OpenSSL_cleanup is getting called (after closing all client sockets)
- binding to just IPv6, just IPv4 or both
- tried making the listen socket blocking and non-blocking
- tried setting SO_LINGER off and also 1 second, both for the listen and peer sockets
The strangest thing is this app has been working fine for more than a decade, but just this week I started seeing this behavior across multiple servers, and multiple versions of Windows. I noticed it when I upgraded to the latest OpenSSL version (3.5.1), but I have since reverted to the previous version (3.0.15) but that didn't help. OpenSSL only works on the peer sockets anyway - it doesn't handle the accept() call, so it never touches the listen socket. The code is C/C++ - not .NET.
My main question is why is System shown listening on my port after I close the socket, and how can I prevent it?