Normally, we code it like this:
while True:
job = queue.get()
...
But is it also possible to do something along the lines of:
for job in queue.get():
# Do stuff to job
The real reason why I want to do that is because I wanted to use python-progressbar's auto detect maxval. They do it like for this in progressbar(that):
You can use iter with callable. (You should pass two arguments, one for the callable, the other for the sentinel value)
for job in iter(queue.get, None): # Replace `None` as you need.
# do stuff with job
NOTE This will block when no elements remain and no sentinel value is put. Also, like a while-get loop and unlike normal for loops over containers, it will remove items from the queue.
None is common value, so here's a sample with more concrete sentinel value:
sentinel = object()
for job in iter(queue.get, sentinel):
# do stuff with job
join? Could you update the code in the question?
None in the answer code) after the normal job related values. queue.put(None)NOTE part. It's same for Python 2.x.
I would say this is an easy way to iterate over the queue in some points:
from queue import Queue
q = Queue()
q.put(1)
q.put(2)
q.put(3)
for i in q.queue:
print(i)
queue.mutex for this, but it's not documented so I'm not 100% sure.
queue.Queue is automatically thread-safe, and mp.Queue is automatically process safe. Indeed, using a queue to provide sequencing is a time-tested way of writing concurrent programs that are easily shown to be correct.
For that kind of queue, actually I would not typically use this check of queue.empty(), because I always use it in a threaded context and thus cannot know whether another thread would put something in there in a few milliseconds (thus that check would be useless anyway). I never check a queue for being empty. I rather use a sentinel value which marks the ending of a producer.
So using the iter(queue.get, Sentinel) is more what I like.
If you know that no other thread will put items in the queue anymore and just want to drain it from all currently contained items, then you can use something like this:
class Drainer(object):
def __init__(self, q):
self.q = q
def __iter__(self):
while True:
try:
yield self.q.get_nowait()
except queue.Empty: # On Python 2, use Queue.Empty
break
for item in Drainer(q):
print(item)
or
def drain(q):
while True:
try:
yield q.get_nowait()
except queue.Empty: # On Python 2, use Queue.Empty
break
for item in drain(q):
print(item)
AttributeError: class Queue has no attribute 'Empty'
import Queue use Queue.Empty. Or use from Queue import Empty and then only expect Empty to be thrown.Queue and depending on your import statement you are referring to the one or the other.
My first thought was for the iter function, but the built-in queue module doesn't return a sentinel, so a good alternative might be to define your own wrapper class:
import Queue
class IterableQueue():
def __init__(self,source_queue):
self.source_queue = source_queue
def __iter__(self):
while True:
try:
yield self.source_queue.get_nowait()
except Queue.Empty:
return
This iterator wraps the queue and yields until the queue is empty, then returns, so now you can do:
q = Queue.Queue()
q.put(1)
q.put(2)
q.put(3)
for n in IterableQueue(q):
print(n)
Output:
1
2
3
This method is a bit verbose. It would be interesting if anyone knows anything better using the builtins.
__init__ method.
queueare you using?