Shared memory parallel processing in python

Question

I have a list of strings and I want to bucket these list elements into appropriate buckets.

def bucketElements(i):
    global buckA, buckB, buckC
    if i.startswith('A'):
        buckA.add(i)
    elif i.startswith('B'):
        buckB.add(i)
    elif i.startswith('C'):
        buckC.add(i)

Now I want to call this method in parallel for each element inside my List. Something like this,

buckA,buckB, buckC = set(), set(), set()
pool = multiprocessing.Pool(processes=10)
pool.map(bucketElements, buckList)

Since I'm updating global variables inside my function I cannot use multiprocessing. is there anyway I can improve my processing? currently I'm calling it like this,

buckA,buckB, buckC = set(), set(), set()
for i in buckList:
    bucketElements(i)

sorry my bad i mentioned as list of integers, it is actually list of strings buckList = ['Adasdas', 'Asdsd', 'Xsadsad', 'Ssdsd', 'Bdasdsds'] — MikA
– MikA, Commented Jun 22, 2017 at 12:54

stovfl · Accepted Answer · 2017-06-22 19:30:42Z

1

You have 3 Options:

Queue()
Manager.Value()
Manager.list()

Python » Documentation: multiprocessing.html#managers

answered Jun 22, 2017 at 19:30

stovfl

15.6k7 gold badges26 silver badges54 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Shared memory parallel processing in python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related