1

so I have Queue.Queue()

i have bunch of producers who puts jobs into that Queue.Queue() and bunch of consumers who pops from the queue

1) is there benefit of capping the Queue size vs. not doing so?
2) by not capping the size, does it really not have any size limit? can grow forever?

I've noticed that deadlock seems to occur more when queue has a fixed size

Improve this question

1 Answer 1

Reset to default
1

If you don't cap the size, the Queue can grow until you run out of memory. So no size limit is imposed by Python, but your machine still has finite resources.

In some applications (probably most), the programmer knows memory consumption can't become a problem, due to the specific character of their application.

But if, e.g., you have producers that "run forever", and consumers that run much slower than producers, capping the size is essential to avoid unbounded memory demands.

As to deadlocks, it's highly unlikely that the implementation of Queue is responsible for a deadlock regardless of whether the Queue's size is bounded; far more likely that deadlocks are due to flaws in application code. For example, picture a producer that fetches things over a network, and mistakenly suppresses errors when the network connection is broken. Then the producer can fail to produce any new items, and so a consumer will eventually block forever waiting for something new to show up on the Queue. That "looks like" deadlock, but the cause really has nothing to do with Queue.

Improve this answer
Sign up to request clarification or add additional context in comments.

9 Comments

but there is a correlation (possibly causation) when i played with Queue.Queue() size i.e. i've started with 100, and all the way to 5000. deadlock was way more consistnt at 100. maybe capping the size revealed something that uncapped didnt?
I believe you, but that changes nothing about what I already said ;-) It's still far more likely due to application logic flaws than a bug in Queue. The Queue code has been stable for years and used by countless thousands of applications with no "deadlock bug" ever reported. Indeed, the Queue implementation is so simple that it's darned near "obviously correct" by inspection.
@ealon, nobody can guess what your code is doing. If you need help with a deadlock, your best chance is to post minimal, runnable code so others can see what you're doing and reproduce the behavior themselves.
i understand. i wasnt blaming Queue per-say for deadlock. I was wondering if there is some behavior difference between capped Queue and uncapped Queue. when Queue is at its maxsize it blocks until more items can put in where as uncapped wouldnt hit this arbitrary block. wonder if that does anything to my code
stackoverflow.com/questions/39669463/…
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.