Send and return variable with c# API call?

Question

I have a c# script task in an ssis package designed to geocode data through my company's proprietary system. It currently works like this:

1) Pull query of addresses and put in data table 2) Loop through that table and Foreach row, build request, send request, wait for response, then insert back into the database.

The issue is that each call takes forever to return, because before going out and getting a new address on the api side, it checks a current database(string match) to ensure the address does not already exist. If not exists, then go out and get me new data from a service like google.

Because I'm doing one at a time, it makes it easy to keep the ID field with the record when I go back to insert it into the database.

Now comes the issue at hand... I was told to configure this as multi-thread or asynchronous. Here is the page I was reading on here about this topic: ASP.NET Multithreading Web Requests

   var urls = new List<string>();

    var results = new ConcurrentBag<OccupationSearch>();

    Parallel.ForEach(urls, url =>
    {
        WebRequest request = WebRequest.Create(requestUrl);

        string response = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();

        var result = JsonSerializer().Deserialize<OccupationSearch>(new JsonTextReader(new StringReader(response)));

        results.Add(result);
    });

Perhaps I'm thinking about this wrong, but if I send 2 requests(A & B) and lets say B actually returns first, how can I ensure that when I go back to update my database I'm updating the correct record? Can I send the ID with the API call and return it?

My thoughts are to create an array of requests, burn through them without waiting for a response and return those value in another array, that I will then loop through on my insert statement.

Is this a good way of going about this? I've never used Parrallel.ForEach, and all the info I find on it is too technical for me to visualize and apply to my situation.

It looks like this code sends exactly the same request twice so you should get exactly the same answer twice. So who cares what order A & B are returned if it is the same response. Has any body actually tested this code? you have a list urls but then it is never used in the ForEach. So what message does this code actual sends. — jdweng
– jdweng, Commented Oct 12, 2017 at 23:49

John Wu · Accepted Answer · 2017-10-12 23:47:33Z

0

Perhaps I'm thinking about this wrong, but if I send 2 requests(A & B) and lets say B actually returns first, how can I ensure that when I go back to update my database I'm updating the correct record? Can I send the ID with the API call and return it?

None of your code contains anything that looks like an "ID," but I assume everything you need is in the URL. If that is the case, one simple answer is to use a Dictionary instead of a Bag.

    List<string> urls = GetListOfUrlsFromSomewhere();

    var results = new ConcurrentDictionary<string, OccupationSearch>();

    Parallel.ForEach(urls.Distinct(), url =>
    {
        WebRequest request = WebRequest.Create(url);

        string response = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();

        var result = JsonSerializer().Deserialize<OccupationSearch>(new JsonTextReader(new StringReader(response)));

        results.TryAdd(url, result);
    });

After this code is done, the results dictionary will contain entries that correlate each response back to the original URL.

Note: you might want to use HttpClient instead of WebClient, and you should take care to dispose of your disposable objects, e.g. StreamReader and StringReader.

edited Oct 12, 2017 at 23:47

answered Oct 12, 2017 at 23:40

John Wu

52.5k8 gold badges50 silver badges92 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user3486773 Over a year ago

This is not my code, this was from the link. I actually do use HTTPClient. Also I realized the easiest solution is simply to convert my Foreach loop to a Parallel.Foreach loop. That way it won't matter about ID's etc...

Collectives™ on Stack Overflow

Send and return variable with c# API call?

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related