0

On Windows Server 2022 I'm running a custom HTTPS server (part of a larger app). The service binds, listens and then accepts on port 81 for a while and netstat -abn shows:

TCP    0.0.0.0:81             0.0.0.0:0              LISTENING 
[MySvc.exe]

All is well. Then after some time the process needs to close so instead of calling accept() again, it instead calls closesocket(). I'm 100% sure it is calling closesocket() on the listening socket (stepped through the debugger, and looked at logs showing socket value).

Not every single time, but frequently I'll see this with netstat:

TCP    0.0.0.0:81             0.0.0.0:0              LISTENING
[System]

The socket was closed and the process is gone, but the System process is "listening" on my port now!

Now the problem. When the service starts again, I end up with:

TCP    0.0.0.0:81             0.0.0.0:0              LISTENING 
[MySvc.exe] 
TCP    0.0.0.0:81             0.0.0.0:0              LISTENING  
[System]

This happens because I set SO_REUSEADDR. If I don't then my service can't bind to the port at all. The problem is now when my service calls accept(), it never gets any connections as long as System is listening. All return codes are checked to bind(), listen() and accept(). If accept is called on a blocking socket, it doesn't return even when clients are attempting to connect.

After maybe 10-30 minutes System stops listening on the port and everything starts working (while my application is running and waiting for accept() to starting working, which it does after the 10-30 minutes).

Things I've checked:

  1. peer sockets (returned from accept) are closed as much as possible (they are given to OpenSSL, but still tracked and closed). When the process stops there are usually some in TIME_WAIT or maybe CLOSE_WAIT, so I think they were all closed 1.a. making sure to call SSL_shutdown and SSL_free, but also tried without it
  2. tried just terminating the process with TerminateProcess() with no closesocket() on anything
  3. made sure WSACleanup is called the same number of times as WSAStartup, also more times, and also not at all
  4. made sure OpenSSL_cleanup is getting called (after closing all client sockets)
  5. binding to just IPv6, just IPv4 or both
  6. tried making the listen socket blocking and non-blocking
  7. tried setting SO_LINGER off and also 1 second, both for the listen and peer sockets

The strangest thing is this app has been working fine for more than a decade, but just this week I started seeing this behavior across multiple servers, and multiple versions of Windows. I noticed it when I upgraded to the latest OpenSSL version (3.5.1), but I have since reverted to the previous version (3.0.15) but that didn't help. OpenSSL only works on the peer sockets anyway - it doesn't handle the accept() call, so it never touches the listen socket. The code is C/C++ - not .NET.

My main question is why is System shown listening on my port after I close the socket, and how can I prevent it?

1

1 Answer 1

0

When a TCP connection is closed, the OS of the "active" closing peer (the peer who initiated the close) keeps the connection active for an extra period of time in order to catch any delayed packets off the network. This is known as the TIME_WAIT state.

So, if the server is the one actively closing its connections (which is common in HTTP), the connections enter TIME_WAIT on the server side, and if the server needs to restart its listening then the listening port may still be in-use and blocked without SO_REUSEADDR. Ideally, you want the clients to be active closers so they enter TIME_WAIT instead of the server.

See TIME_WAIT and its design implications for protocols and scalable client server systems

Sign up to request clarification or add additional context in comments.

3 Comments

Agree with all of the above. However, the listening port is not entering TIME_WAIT when closed - it stays in LISTENING as shown above.
@DougN I didn't say the listening port enters TIME_WAIT. Ports don't enter TIME_WAIT, connections do. But, the port is in use by connections, so if the server needs to restart and there are old connections still in TIME_WAIT, then the listening port is in use and can't be reopened without SO_REUSEADDR, as you already saw.
The connection that was listening on port 81 was closed. But at that same moment System (PID 4) created a connection that listens on that port (and does for 28 minutes according to recent logs I looked at). Why is a listen port still in use when closed? There are many separate connections shown in netstat with (for example) TCP (local IP):81 (remote IP):51636 TIME_WAIT so THOSE connections will timeout in 2 minutes. But the listening port should not be listening - it was closed. Whatever is happening prevents a server application from restarting which isn't n

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.