How to fetch API data using WebClient and multiple threads?

Question

So I am trying to query an API that's accessible via HTTP ( no authorization ). To speed things up, I tried to use a Parallel.ForEach loop but it seems like the longer it runs, the more errors pop up.

It fails to retrieve more and more requests. I know the API provider isn't limiting me because I can request the very same blocked URLs in my Internet browser. Also, these are different failed URLs each time, so it doesn't seem to be the case of malformed requests.

The error doesn't seem to occur while I use single threaded foreach loop.

My malfunctioning loop is below:

Parallel.ForEach(this.urlArray, singleUrl => {
this.apiResponseBlob = new System.Net.WebClient ().DownloadString(singleUrl );
this.responsesDictionary.Add(singleUrl, apiResponseBlob);
}

Normal foreach loop works fine but is very slow:

foreach (string singleUrl in this.urlArray) {
this.apiResponseBlob = new System.Net.WebClient ().DownloadString(singleUrl);
this.responsesDictionary.Add(singleUrl, apiResponseBlob);
}

Also: I've had a solution in PHP - I spawned several "fetchers" simultaneously and it never hung up. It seems strange to me that PHP would handle multithreaded retrieval better than C# so I must obviously miss something.

How do I query the API fastest way? Without these strange failures?

Wouldn't it be easier to use the async version of that call? — rene
– rene, Commented Dec 8, 2014 at 12:44
You mean together with Parrarel.ForEach or normal ForEach loop? — gggggggg5555
– gggggggg5555, Commented Dec 8, 2014 at 12:58
with the normal foreach and let the WbClient instance handle the completion — rene
– rene, Commented Dec 8, 2014 at 12:59

Community · Accepted Answer · 2017-05-23 11:59:56Z

2

Hi did you try to speed up your code with a sync downloads like in this question (see marked answer):

DownloadStringAsync wait for request completion

your could loop through your uris and get a callback for each successfull download.

EDIT : i have seen that you use

this.apiResponseBlob = DL

when you use multithreading every thread tries to write in that variable. This could be a reason vor your bug. Try using an instance of that object type or use

lock{}

so that only one thread can write this variable at time. http://msdn.microsoft.com/de-de/library/c5kehkcz.aspx

like

    Parallel.ForEach(this.urlArray, singleUrl => {
    var apiResponseBlob = new System.Net.WebClient ().DownloadString(singleUrl );
    lock(singleUrl.ToString()){
    this.responsesDictionary.Add(singleUrl, apiResponseBlob);
}
    }

edited May 23, 2017 at 11:59

CommunityBot

11 silver badge

answered Dec 8, 2014 at 12:46

Bjego

6835 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

gggggggg5555 Over a year ago

What about when the WebClient is inside other class? Can I still use async by passing this class just one url?

Bjego Over a year ago

yes you can. but you have to use an event - for the response. Something like this. e.g. (written from my mind ;) not proofed public class MyDownloader{ public event EventHandler<EventArgs> DlFinished; public void DLAsync(uri url){ var client = new WebClient(); client.DownloadStringCompleted += (sender, e) => { doSomeThing(e.Result); this.DlFinished(null,null); }; client.DownloadStringAsync(uri); }} usage from other class: MyDownloader loader = new MyDownloader; loader.DlFinished+=CallbackFunction; loader.DlAsync(uri);

Bjego Over a year ago

hope this is readable ;) more info here about custom events: stackoverflow.com/questions/6644247/simple-custom-event

Collectives™ on Stack Overflow

How to fetch API data using WebClient and multiple threads?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related