Multiple HTTP request in C#

Question

I need to send about 200 HTTP requests in parallel to different servers and get response. I use HttpWebRequest class in C#. But I don't see good time enhancement when requests are handled in parallel. For example if one request needs 3sec to get response, 2 request in parallel - 6sec, 3 requests - 8 secs, 4 requests - 11sec ... It is not good, I hope that there should be best time, about 10 sec for 200 requests. It looks like only 2-3 requests performs in parallel, but timeout starts immediately after WebRequest object creation. I tried set DefaultConnectionLimit and MaxServicePoints values, but id didn't help. As I understand these parameters for number of requests to one site in parallel. I need requests to different sites.

Code example that I use to test:

ArrayList a = new ArrayList(150);

for (i = 50; i < 250; i++ )
{
   a.Add("http://207.242.7." + i.ToString() + "/");
}

for (i = 0; i < a.Count; i++)
{
    Thread t = new Thread(new ParameterizedThreadStart(performRequest));
    t.Start(a[i]);
}


static void performRequest(object ip)
{
      HttpWebRequest req = (HttpWebRequest)WebRequest.Create((stirng)ip);

      HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
}

Сan anyone ever encountered such a problem? Thank you for any suggestions.

I have similar issue myself, it seems that WebRequest.Create() reuse the same TCP connection for some reason. I'm wondering if you found out what the answer to your question is? — ala
– ala, Commented Oct 19, 2011 at 1:20

Jakob Christensen · Accepted Answer · 2010-06-04 09:51:56Z

3

Instead of starting up your own threads try using the asynchronous methods of HttpWebRequest such as HttpWebRequest.BeginGetResponse and HttpWebRequest.BeginGetRequestStream.

answered Jun 4, 2010 at 9:51

Jakob Christensen

15k2 gold badges53 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Nikita Over a year ago

I already have multithread application and need to execute this requests in already creaated threads. This is only example code. Point is that when threads peforms WebRequest.Create and GetResponse actions porfomance is very very slow. Also before HTTP request I perform TCP connection in the same threads on sever's 80 port to know port is opened or closed and it works very fast.

ChoccyButton · Accepted Answer · 2010-06-04 09:52:49Z

3

This might help - http://social.msdn.microsoft.com/forums/en-US/netfxnetcom/thread/1f863f20-09f9-49a5-8eee-17a89b591007

Suggests there is a limit on the number of TCP connections allowed, but that you can increase the limit

answered Jun 4, 2010 at 9:52

ChoccyButton

1159 bronze badges

3 Comments

ChoccyButton Over a year ago

More details here as well - arnosoftwaredev.blogspot.com/2006/09/… This is almost certainly the problem your hitting

Patrick Over a year ago

Although.. your links seems to reference simultaneous connections to one host, and from the example in the question, the OP is making connections to several different hosts... Is this the case or does .NET have a maximum connections limit to any host as well?

ChoccyButton Over a year ago

@Patrick Think you have a point. Depends I guess if all 200 of his requests go to different servers or say 10 to first server, 20 to next, 5 to next, etc. This would hit the problem. Otherwise it might not be the problem. Worth a try at least

k_b · Accepted Answer · 2010-06-04 09:50:42Z

0

Use asynchronous web requests in stead.

Edit: http://msdn.microsoft.com/en-us/library/86wf6409(VS.71).aspx

answered Jun 4, 2010 at 9:50

k_b

2,49018 silver badges9 bronze badges

Comments

Xstahef · Accepted Answer · 2010-06-04 09:59:00Z

0

you can try this :

try
        {
            List<Uri> uris = new List<Uri>();
            uris.Add(new Uri("http://www.google.fr"));
            uris.Add(new Uri("http://www.bing.com"));

            Parallel.ForEach(uris, u =>
           {
               WebRequest webR = HttpWebRequest.Create(u);
               HttpWebResponse webResponse = webR.GetResponse() as HttpWebResponse;
           });
        }
        catch (AggregateException exc)
        {
            exc.InnerExceptions.ToList().ForEach(e =>
                {
                    Console.WriteLine(e.Message);
                });
        }

answered Jun 4, 2010 at 9:59

Xstahef

6541 gold badge6 silver badges14 bronze badges

Comments

Prashant Pimpale · Accepted Answer · 2019-03-27 05:53:33Z

-3

This is the code of android application of sending the request. How can we use the upper code like:

params.add(new BasicNameValuePair("key", "value");
HttpPost request = new HttpPost();
List<NameValuePair> params = new ArrayList<NameValuePair>();
params.add(new BasicNameValuePair("key", "value");// How can we give the value in this format format
post.setEntity(new UrlEncodedFormEntity(params));

httpClient.execute(request);

edited Mar 27, 2019 at 5:53

Prashant Pimpale

10.7k10 gold badges52 silver badges88 bronze badges

answered Feb 14, 2014 at 11:05

user2655792

12 bronze badges

Collectives™ on Stack Overflow

Multiple HTTP request in C#

5 Answers 5

1 Comment

3 Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

1 Comment

3 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related