Parallel.ForEach with adding to list

Question

I'm trying to run multiple functions that connect to a remote site (by network) and return a generic list. But I want to run them simultaneously.

For example:

public static List<SearchResult> Search(string title)
{
    //Initialize a new temp list to hold all search results
    List<SearchResult> results = new List<SearchResult>();

    //Loop all providers simultaneously
    Parallel.ForEach(Providers, currentProvider =>
    {
        List<SearchResult> tmpResults = currentProvider.SearchTitle((title));

        //Add results from current provider
        results.AddRange(tmpResults);
    });

    //Return all combined results
    return results;
}

As I see it, multiple insertions to 'results' may happend at the same time... Which may crash my application.

How can I avoid this?

It would have to be at least .Net 4; Parallel was introduced there. — Matt Mills
– Matt Mills, Commented Nov 3, 2011 at 20:58

Mark Byers · Accepted Answer · 2011-11-03 21:16:36Z

178

You can use a concurrent collection.

The System.Collections.Concurrent namespace provides several thread-safe collection classes that should be used in place of the corresponding types in the System.Collections and System.Collections.Generic namespaces whenever multiple threads are accessing the collection concurrently.

You could for example use ConcurrentBag since you have no guarantee which order the items will be added.

Represents a thread-safe, unordered collection of objects.

edited Nov 3, 2011 at 21:16

answered Nov 3, 2011 at 20:58

Mark Byers

844k202 gold badges1.6k silver badges1.5k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

lkg Over a year ago

Yes, this is the actual answer. You'll get better performance (generally) with concurrent collections.

Theodor Zoulias Over a year ago

A ConcurrentQueue<T> might be preferable to a ConcurrentBag<T>, because the former preserves the insertion order. The ConcurrentBag<T> is an extremely specialized collection intended for mixed producer consumer scenarios, and adding items in parallel (without taking items from the collection) is not such a scenario.

Haedrian · Accepted Answer · 2013-12-03 21:13:51Z

74

//In the class scope:
Object lockMe = new Object();    

//In the function
lock (lockMe)
{    
     results.AddRange(tmpResults);
}

Basically a lock means that only one thread can have access to that critical section at the same time.

edited Dec 3, 2013 at 21:13

answered Nov 3, 2011 at 20:57

Haedrian

4,3202 gold badges36 silver badges54 bronze badges

7 Comments

shaharmor Over a year ago

But what will happend if WHILE those results are being added the results from another provider try to add to? will they FAIL or WAIT until possible?

Haedrian Over a year ago

When there's a lock, the thread will wait until it can get the lock.

Henk Holterman Over a year ago

A minor point: this is not the safest choice for the lock object. Better to use a special private object: lock(resultsLock) .

Piotr Kula Over a year ago

locks can slow down the overall execution time though.. concurrent collections seem better to avoid that

Frank Q. Over a year ago

Why do we need a lock at class level if the object modified is declared locally ? If we lock(results) and then add to it what is wrong ? Lock at class level means, if multiple requests are calling Search then you have degraded the performance

|

Matas Vaitkevicius · Accepted Answer · 2015-12-01 10:17:03Z

44

For those who prefer code:

public static ConcurrentBag<SearchResult> Search(string title)
{
    var results = new ConcurrentBag<SearchResult>();
    Parallel.ForEach(Providers, currentProvider =>
    {
        results.Add(currentProvider.SearchTitle((title)));
    });

    return results;
}

answered Dec 1, 2015 at 10:17

Matas Vaitkevicius

61.9k37 gold badges253 silver badges278 bronze badges

1 Comment

Anthony McGrath Over a year ago

Have to use a loop: foreach (var item in currentProvider.SearchTitle((title))) results.Add(item);

Community · Accepted Answer · 2020-06-20 09:12:55Z

27

The Concurrent Collections are new for .Net 4; they are designed to work with the new parallel functionality.

See Concurrent Collections in the .NET Framework 4:

Before .NET 4, you had to provide your own synchronization mechanisms if multiple threads might be accessing a single shared collection. You had to lock the collection ...

... the [new] classes and interfaces in System.Collections.Concurrent [added in .NET 4] provide a consistent implementation for [...] multi-threaded programming problems involving shared data across threads.

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Nov 3, 2011 at 20:58

Matt Mills

8,8826 gold badges44 silver badges65 bronze badges

Comments

Douglas · Accepted Answer · 2017-06-08 18:33:56Z

17

This could be expressed concisely using PLINQ's AsParallel and SelectMany:

public static List<SearchResult> Search(string title)
{
    return Providers.AsParallel()
                    .SelectMany(p => p.SearchTitle(title))
                    .ToList();
}

edited Jun 8, 2017 at 18:33

answered Nov 16, 2014 at 21:19

Douglas

55.1k14 gold badges145 silver badges193 bronze badges

2 Comments

Kugan Kumar Over a year ago

linq selectMany is great, sadly linq is slower than normal foreach. :(

Douglas Over a year ago

Don't micro-optimize. The OP implied that SearchTitle connects to a remote site. Its latency will be several orders of magnitude slower than the difference between LINQ and foreach.

Collectives™ on Stack Overflow

Parallel.ForEach with adding to list

5 Answers 5

2 Comments

7 Comments

1 Comment

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

2 Comments

7 Comments

1 Comment

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related