4

I need to convert a threading application to a multiprocessing application for multiple reasons (GIL, memory leaks). Fortunately the threads are quite isolated and only communicate via Queue.Queues. This primitive is also available in multiprocessing so everything looks fine. Now before I enter this minefield I'd like to get some advice on the upcoming problems:

  1. How to ensure that my objects can be transfered via the Queue? Do I need to provide some __setstate__?
  2. Can I rely on put returning instantly (like with threading Queues)?
  3. General hints/tips?
  4. Anything worthwhile to read apart from the Python documentation?
Improve this question
2
  • Initializing database connections in subclass(Process).__init__ is a problem. :-/ Commented Nov 4, 2010 at 14:48
  • 1
    another user recently rewrote my threaded example with the multiprocessing library. i was using queue as well. might be worth a look: stackoverflow.com/questions/4072816/… Commented Nov 9, 2010 at 7:37

1 Answer 1

Reset to default
5

Answer to part 1:

Everything that has to pass through a multiprocessing.Queue (or Pipe or whatever) has to be picklable. This includes basic types such as tuples, lists and dicts. Also classes are supported if they are top-level and not too complicated (check the details). Trying to pass lambdas around will fail however.

Answer to part 2:

A put consists of two parts: It takes a semaphore to modify the queue and it optionally starts a feeder thread. So if no other Process tries to put to the same Queue at the same time (for instance because there is only one Process writing to it), it should be fast. For me it turned out to be fast enough for all practical purposes.

Partial answer to part 3:

  • The plain multiprocessing.queue.Queue lacks a task_done method, so it cannot be used as a drop-in replacement directly. (A subclass provides the method.)
  • The old processing.queue.Queue lacks a qsize method and the newer multiprocessing version is inaccurate (just keep this in mind).
  • Since filedescriptors normally inherited on fork, care needs to be taken about closing them in the right processes.
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.