System.Net.Http.HttpRequestException on web server

Question

I am hosting an asp.net web api on an azure web app. The app is hosted on 2 side by side instances of azure web apps.

However recently our peak requests have increased, and we have started seeing the following error in our logs:

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full

In our web server we make many short lived http post requests to additional servers using HttpClinet. From my research of the above error code, it seems that the underlying OS is running out of 'user' TCP ports. I first thought this was some bug with the HttpClient .net library. However we have tried a static instance of HttpClient as well as a using dispose method. Neither resolve our issue. The only thing that helps is us manually restarting the web app every hour, which is definitely not a long term solution.

I am unable to run 'netstat' on the azure web app to see what is happening with the sockets.

The number requests we are doing is roughly 100 000 a day. This should not be overloading one machine, never mind two. So something must be very wrong.

How can we resolve this issue?

I have some questions:

Is httpclient up for the job, the job being thousands of http post requests?
Is there anyway to force HttpClient to use http2 where it will do multiple requests on a single tcp socket.
What other technology options do have? Essentially we want to send a json object from one server to another and get a json object back as a response. Would something like azure service bus work where it would have several sockets and push many requests over those sockets?

We can only guess. There is no code and only a partial error message. — Crowcoder
– Crowcoder, Commented May 19, 2019 at 16:50
I don't believe code is the issue here.. hence why I have not pasted it. — Zapnologica
– Zapnologica, Commented May 19, 2019 at 17:33
There is even less information about the application than there is about the code. You build it, have access to the code, the servers, and you don't know the problem - how is anyone else going to know? — Crowcoder
– Crowcoder, Commented May 19, 2019 at 17:38

Mobrine Hayde · Accepted Answer · 2019-05-20 23:32:30Z

1

I just came across the issue and it seems that when you reach TCP ports greater than 5000 that error is raised giving the same exception as you have (if you send 100000 request a day, it is only natural to open more than 5000 port in one time)

There is also a way to fix it that is to add MaxUserPort (DWORD value type) in Registry Editor at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

for more info about this issue https://support.microsoft.com/en-us/help/196271/when-you-try-to-connect-from-tcp-ports-greater-than-5000-you-receive-t

Also changing MaxUserPort may fix your issue, but it will create another one, especially if you receive a TCP attack by opening a lot of ports to crash your system, so it's best to choose a maximum value with caution (and that's why there is this setting)

Notice : you can also use Tcp client, but then you need to manually build the http request, it is best to stick with http client

edited May 20, 2019 at 23:32

answered May 20, 2019 at 23:08

Mobrine Hayde

3651 gold badge2 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Zapnologica Over a year ago

Thanks for this response, I donot believe I can edit those values when using azure web app. But maybe it might be worth while looking into our own implementation of http client using a tcp client.

Karthikeyan VK · Accepted Answer · 2022-06-14 03:47:57Z

0

I restarted my web app three times, stopped & started once and it started working. Not sure what is this weird issue. I stopped the service for more than 12 hours and we had this issue.

answered Jun 14, 2022 at 3:47

Karthikeyan VK

6,0964 gold badges45 silver badges55 bronze badges

Collectives™ on Stack Overflow

System.Net.Http.HttpRequestException on web server

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related