python - multiprocessing with queue

Question

Here is my code below , I put string in queue , and hope dowork2 to do something work , and return char in shared_queue

but I always get nothing at while not shared_queue.empty()

please give me some point , thanks.

import time
import multiprocessing as mp


class Test(mp.Process):

    def __init__(self, **kwargs):
        mp.Process.__init__(self)
        self.daemon = False
        print('dosomething')

    def run(self):
        manager = mp.Manager()
        queue = manager.Queue()
        shared_queue = manager.Queue()
        # shared_list = manager.list()
        pool = mp.Pool()
        results = []
        results.append(pool.apply_async(self.dowork2,(queue,shared_queue)))
        while True:
            time.sleep(0.2)
            t =time.time()
            queue.put('abc')
            queue.put('def')
            l = ''
            while not shared_queue.empty():
                l = l + shared_queue.get()
            print(l)
            print( '%.4f' %(time.time()-t))

        pool.close()
        pool.join()

    def dowork2(queue,shared_queue):
        while True:
            path = queue.get()
            shared_queue.put(path[-1:])

if __name__ == '__main__':
    t = Test()
    t.start()
    # t.join()
    # t.run()

Do you expect a result as well from your dowork2 or just the feedback in shared_queue? — Hannu
– Hannu, Commented Jan 26, 2018 at 15:36

Hannu · Accepted Answer · 2018-01-26 15:50:00Z

1

I managed to get it work by moving your dowork2 outside the class. If you declare dowork2 as a function before Test class and call it as

results.append(pool.apply_async(dowork2, (queue, shared_queue)))

it works as expected. I am not 100% sure but it probably goes wrong because your Test class is already subclassing Process. Now when your pool creates a subprocess and initialises the same class in the subprocess, something gets overridden somewhere.

Overall I wonder if Pool is really what you want to use here. Your worker seems to be in an infinite loop indicating you do not expect a return value from the worker, only the result in the return queue. If this is the case, you can remove Pool.

I also managed to get it work keeping your worker function within the class when I scrapped the Pool and replaced with another subprocess:

foo = mp.Process(group=None, target=self.dowork2, args=(queue, shared_queue))
foo.start()
# results.append(pool.apply_async(Test.dowork2, (queue, shared_queue)))
while True:
    ....

(you need to add self to your worker, though, or declare it as a static method:)

def dowork2(self, queue, shared_queue):

answered Jan 26, 2018 at 15:50

Hannu

12.4k4 gold badges38 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Relax ZeroC Over a year ago

Thank you very much , @Hannu . You save my life :)

Collectives™ on Stack Overflow

python - multiprocessing with queue

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related