Superimpose async webrequest with database retrieval

Question

In the system I'm currently building I have to make webrequests to an API which provides a calculation service. This service requires a set of complex parameters which I have to retrieve from my database. Currently I'm using entity framework for retrieving these entities and for each entity I'm making a request to this api, retrieve the result and at the end save all results to the database (everything done synchronously)

There will be scaling issues with this approach when the set of entities increases (since I have to call the calculation service every 30 minutes on each entity). Because of this I would like to make the database retrieval and web request for an entity in parallell (or async) with the same operations for other entities. (Not with the purpose of reducing time for data loading but to do work while waiting for the webrequest to complete)

Since EF 5 context is not thread safe, what is my best alternatives for achieving this? Should I write specific SQL queries, use LINQ etc? Does anyone have code examples for a similar approach (db retrieval for webrequest in parallell)

EDIT Adding a small code sample (very simplified). Assuming that the call to the webservice may take a couple of seconds this will not scale.

foreach(entityId in entityIds)
{
    var entity = _repository.Find(entityId);
    _repository.LoadData(entity);
    _validator.ValidateData(entity);
    var result = _webservice.Call(entity);
    entity.State = result.State;
}

_repository.SaveChanges();

You can still make the async call to the web service while calling the database. Or use Task.Run to call the database if necessary. Can you provide some code for what you want to do? — Paulo Morgado
– Paulo Morgado, Commented May 27, 2013 at 21:00

svick · Accepted Answer · 2013-05-27 16:57:35Z

1

What you could do is to use a producer-consumer architecture: One thread accesses the database and adds the data to something like BlockingCollection. Another thread (or multiple threads) reads the data from the collection and performs the web request.

answered May 27, 2013 at 16:57

svick

247k54 gold badges407 silver badges535 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

olif Over a year ago

Hi. Sorry, perhaps a stupid question but by using a single thread for data retrieval means I can continue using EF for reading data? If so, I guess I could use the same pattern for saving the result from the web request as well?

svick Over a year ago

@user842077 Yeah, that's the point of this approach, that you can continue using types that are not thread-safe.

Paulo Morgado · Accepted Answer · 2013-05-28 10:32:32Z

0

There are different ways for you to parallelize this. It all depends on what you really want/need.

One way would be to use a Paralle.ForEach.

Parallel.ForEach(
    entityIds,
    entityId =>
    {
        var entity = _repository.Find(entityId);
        _repository.LoadData(entity);
        _validator.ValidateData(entity);
        var result = _webservice.Call(entity);
        entity.State = result.State;
    });

_repository.SaveChanges();

answered May 28, 2013 at 10:32

Paulo Morgado

15k3 gold badges36 silver badges64 bronze badges

4 Comments

olif Over a year ago

The entity framework context is not thread safe. Because of this I cannot retrieve data in paralell.

Paulo Morgado Over a year ago

I wasn't aware of this. I'm not an EF expert. So, how would you purpose to do it? How many IDs does entityIds have? What's the size of each entity? One way could be retrieving all entities and then parallelize the calls to the web service. Would this do?

olif Over a year ago

I think the answer provided by svick is the way to go. An alternative would be to throw out EF and go by your approach.

Paulo Morgado Over a year ago

Dropping EF wouldn't get you far because I don't think you could use the same db connection instance. Pretty much the same as using multiple EF contexts. I wonder if the EF context would be thread safe if it not tracking changes.

Collectives™ on Stack Overflow

Superimpose async webrequest with database retrieval

2 Answers 2

2 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related