Passing Queue to Threadpool - Python

Question

Getting confused using pool with list vs queues. I get the error: TypeError: 'int' object is not iterable when passing q.get in place of a list. Hopefully it's just an oversight.

from multiprocessing.dummy import Pool 
import Queue

numbers = [11,12,13,14]

def printNumbers(n):
    print n

q = Queue.Queue()    
for i in (1,2,3,4):
   q.put(i)

#multiThread with list numbers
pool = Pool(4) 
results = pool.map(printNumbers, numbers)
pool.close() 
pool.join() 

#multiThread with queue
pool2 = Pool(4) 
results2 = pool2.map(printNumbers, q.get())
pool2.close() 
pool2.join()

why do you want to use a queue at this point. And btw. be aware between the differences of threading and multiprocesing — Stefan Reinhardt
– Stefan Reinhardt, Commented Feb 4, 2016 at 15:35
Queue as it will be growing as the threads are pulling from it (researching thread vs multiprocessing now). End goal is a script that walks a directory, and creates a fileInfo var used in a pycurl command, then outputs to a log. Right now in order -> parse walk, grab next file, pull info from root path, upload via curl, output to log pass/fail. Hitting huge time delays on the curl so using threads Future goal -> Parse walk, fill queue with file info -> pull from queue with multiple threads doing curl command -> fill logQueue with success/fail -> function pull from logQueue to write out — Tyler
– Tyler, Commented Feb 4, 2016 at 16:01

Community · Accepted Answer · 2017-05-23 12:30:59Z

2

The Queue class does not implement the iterable protocol. You'll need to provide your own Queue implementation that does. Here is an example:

class IterableQueue(Queue.Queue):

    _sentinel = object()

    def __init__(self, maxsize=0):
        Queue.Queue.__init__(self, maxsize)
        self.put(self._sentinel)

    def __iter__(self):
        return iter(self.get, self._sentinel)

You can read the details of __iter__ and how the sentinel is used at https://docs.python.org/2/library/functions.html#iter

Please note that my example queue is not what you want if you want a queue that will block when empty until more items are added. For that, see Python iterable Queue

However, when I used that version with your code, it never terminated. To use that one, you'll need to restructure a bit so that you can call the close method when you know you have no more items to add to the queue.

edited May 23, 2017 at 12:30

CommunityBot

11 silver badge

answered Feb 4, 2016 at 17:24

foundart

3702 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Passing Queue to Threadpool - Python

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related