ASP.NET Multithreading Web Requests

Question

I'm building a page in my ASP.NET solution that retrieves most of its data from a 3rd Party API Service.

The page itself will need to make approximately 5 individual different calls to the API to populate all of its controls, as each Web Request to the API brings back different sets of Data.

I would like to deal with each separate Web request i make on a new thread, simultaneously so that load time is reduced.

Each request I'm making looks like this :-

WebRequest request = WebRequest.Create(requestUrl);

string response = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();

return new JsonSerializer().Deserialize<OccupationSearch>(new JsonTextReader(new StringReader(response)));

Firstly, should i be attempting this? i.e, is it safe? and will it enhance performance by multi-threading this out.

Secondly, what is the best way to go about doing this? There's many different ways to multi-thread systems, but which will suit this task best?

Each web request should already be a different worker thread in the IIS, depending on how much load your service is under, and how your IIS is set up. So if you talk multithreading, you're going to try and subdivide the work done for each request into multiple threads. — Allan S. Hansen
– Allan S. Hansen, Commented May 28, 2014 at 13:08
I'd take a look at doing it asynchronously. Use request.GetResponseAsync() et al. — DavidG
– DavidG, Commented May 28, 2014 at 13:09

Matías Fidemraizer · Accepted Answer · 2014-05-28 13:25:26Z

15

Multi-threading is the right choice, but I would call it doing stuff asynchronously.

Anyway, you should know that multi-threading works different in IIS.

IIS worker process will finish a request once all child threads end, and this is a big problem, because you don't want to hold a worker process for a long time but re-use it for other requests. Actually it's a thread pool.

This is why ASP.NET offers its own approach to implement asynchronity and if you use the right approach, IIS will be able to process more requests at once because asynchronous work will be executed outside IIS process model.

I would suggest you to read more about ASP.NET async:

Conclusion: use asynchronous work and this will make a more efficient use of server resources, but first learn more about how to do it in the right way!

answered May 28, 2014 at 13:25

Matías Fidemraizer

65.1k19 gold badges132 silver badges216 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Matías Fidemraizer Over a year ago

@Derek No problem. In fact, you should go this way because other approaches will end in deadlocks.

Stephen Cleary · Accepted Answer · 2014-05-28 15:03:38Z

Multithreading is not recommended on ASP.NET; because ASP.NET/IIS does its own multithreading, and any multithreading you do will interfere with those heuristics.

What you really want is concurrency, more specifically asynchronous concurrency since your operations are I/O-bound.

The best approach is to use HttpClient with the Task-based Asynchronous Pattern:

public async Task<OccupationSearch> GetOccupationAsync(string requestUrl)
{
  // You can also reuse HttpClient instances instead of creating a new one each time
  using (var client = new HttpClient())
  {
    var response = client.GetStringAsync(requestUrl);
    return new JsonSerializer().Deserialize<OccupationSearch>(new JsonTextReader(new StringReader(response)));
  }
}

This is asynchronous, and you can easily make it concurent by using Task.WhenAll:

List<string> urls = ...;
OccupationSearch[] results = await Task.WhenAll(urls.Select(GetOccupationAsync));

Philip Pittle · Accepted Answer · 2014-05-28 13:21:20Z

3

I would argue that Multithreading (within reason) would provide benefits as the way your code is written know the calls to GetResponse().GetResponseStream() are blocking.

One of the easiest ways to improve performance is to use a Parallel.ForEach:

        var urls = new List<string>();

        var results = new ConcurrentBag<OccupationSearch>();

        Parallel.ForEach(urls, url =>
        {
            WebRequest request = WebRequest.Create(requestUrl);

            string response = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();

            var result = JsonSerializer().Deserialize<OccupationSearch>(new JsonTextReader(new StringReader(response)));

            results.Add(result);
        });

If you are using .NET 4.5 the new approach is to use async/await. Msdn has a pretty extensive article on this very topic here: http://msdn.microsoft.com/en-us/library/hh300224.aspx and http://msdn.microsoft.com/en-us/library/hh696703.aspx

Scott Hanselman also has a good looking Blog Post on this topic: http://www.hanselman.com/blog/TheMagicOfUsingAsynchronousMethodsInASPNET45PlusAnImportantGotcha.aspx

edited May 28, 2014 at 13:21

answered May 28, 2014 at 13:14

Philip Pittle

12.5k8 gold badges69 silver badges127 bronze badges

3 Comments

LB2 Over a year ago

Wouldn't you need to lock(results) since List<T> is not thread-safe (for writes)?

Philip Pittle Over a year ago

Fair point, to be honest I'm not sure if a List would lose anything since it's not reading and writing. But just be safe, I'll update to use a ConcurrentBag.

LB2 Over a year ago

List is writing - it's writing the Add-ition of result. All you'd need is a a lock around it, and that'll do it.

Collectives™ on Stack Overflow

ASP.NET Multithreading Web Requests

3 Answers 3

1 Comment

Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related