How to wait for multiple async http request

Question

I'd like to ask about how to wait for multiple async http requests.

My code is like this :

    public void Convert(XDocument input, out XDocument output)
    {
        var ns = input.Root.Name.Namespace;

        foreach (var element in input.Root.Descendants(ns + "a"))
        {
            Uri uri = new Uri((string)element.Attribute("href"));

            var wc = new WebClient();
            wc.OpenReadCompleted += ((sender, e) =>
            {
                element.Attribute("href").Value = e.Result.ToString();
            }
            );
            wc.OpenReadAsync(uri);
        }

        //I'd like to wait here until above async requests are all completed.
        output = input;  
    }

Dose anyone know a solution for this?

bottlenecked · Accepted Answer · 2012-05-26 16:40:31Z

There is an article by Scott Hanselman in which he describes how to do non blocking requests. Scrolling to the end of it, there is a public Task<bool> ValidateUrlAsync(string url) method.

You could modify it like this (could be more robust about response reading)

public Task<string> GetAsync(string url)
{
    var tcs = new TaskCompletionSource<string>();
    var request = (HttpWebRequest)WebRequest.Create(url);
    try
    {
        request.BeginGetResponse(iar =>
        {
            HttpWebResponse response = null;
            try
            {
                response = (HttpWebResponse)request.EndGetResponse(iar);
                using(var reader = new StreamReader(response.GetResponseStream()))
                {
                    tcs.SetResult(reader.ReadToEnd());
                }                    
            }
            catch(Exception exc) { tcs.SetException(exc); }
            finally { if (response != null) response.Close(); }
        }, null);
    }
    catch(Exception exc) { tcs.SetException(exc); }
    return tsc.Task; 
}

So with this in hand, you could then use it like this

var urls=new[]{"url1","url2"};
var tasks = urls.Select(GetAsync).ToArray();
var completed = Task.Factory.ContinueWhenAll(tasks,
                    completedTasks =>{
                                          foreach(var result in completedTasks.Select(t=>t.Result))
                                          {
                                              Console.WriteLine(result);
                                          }
                                      });
completed.Wait();
//anything that follows gets executed after all urls have finished downloading

Hope this puts you in the right direction.

PS. this is probably as clear as it can get without using async/await

agent-j · Accepted Answer · 2012-05-26 12:47:14Z

Consider using continuation passing style. If you can restructure your Convert method like this,

public void ConvertAndContinueWith(XDocument input, Action<XDocument> continueWith)
{
    var ns = input.Root.Name.Namespace;
    var elements = input.Root.Descendants(ns + "a");
    int incompleteCount = input.Root.Descendants(ns + "a").Count;
    foreach (var element in elements)
    {
        Uri uri = new Uri((string)element.Attribute("href"));

        var wc = new WebClient();
        wc.OpenReadCompleted += ((sender, e) =>
        {
            element.Attribute("href").Value = e.Result.ToString();
            if (interlocked.Decrement(ref incompleteCount) == 0)
               // This is the final callback, so we can continue executing.
               continueWith(input);
        }
        );
        wc.OpenReadAsync(uri);
    }
}

You then run that code like this:

XDocument doc = something;
ConvertAndContinueWith(doc, (finishedDocument) => {
    // send the completed document to the web client, or whatever you need to do
});

Collectives™ on Stack Overflow

How to wait for multiple async http request

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related