python - using multiple processes / parallel processing

Question

So I wrote a small python script that allows me to specify some folder that contains video files, and some output directory, and the program loops over all the video files, and converts them using handbrake like:

proc = subprocess.Popen('HandBrakeCLI -i ... -o ...')
proc.wait()

So it does each file in the directory, one by one. I have a quad core machine, and want to speed things by doing video conversions in parallel but I don't entirely understand how to approach that.

Do I need something like os.fork()? Something more like the multiprocessing module? I come from single threaded javascript land so this relatively new to me.

Thanks for any input!

mgilson · Accepted Answer · 2012-05-22 17:27:07Z

2

Something similar to this should do the trick:

import multiprocessing
...
def wrapper(currentfile):
    #create your commandline -- for the sake of simplicity, I assume that
    #the only thing that changes is the filename you are passing to
    #HandBrakeCLI (and that the filename is the last argument to pass)
    cmd='HandBrakeCLI -i ... -o ... '+currentfile 
    proc = subprocess.Popen(cmd)
    proc.wait()
    return 'foo'

files=os.listdir(path)  #Any way that you build up your list of files is fine
output = multiprocessing.Pool(4).map(wrapper,files) #output is ['foo', 'foo', ..., 'foo']

Of course, this uses a map-like function for it's side-effects which many python people dislike... but I find it to be intuitive enough -- especially if you leave a comment. I also made the function return 'foo' to demonstrate that you can access the return value from the function quite easily.

edited May 22, 2012 at 17:27

answered May 22, 2012 at 17:18

mgilson

312k70 gold badges656 silver badges722 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Spencer Rathbun · Accepted Answer · 2012-05-22 17:48:00Z

0

I'd suggest using the envoy library. Under the hood it uses the threading library to spawn new cmd line programs, so if you use the connect function like so:

import envoy
proc = envoy.connect('HandBrakeCLI -i ... -o ...')
while proc.status_code = None:
    sleep(5)

You can spawn as many as you want at a time, and wait until one exits before spawning another. Please note if you have issues, that I've got a fork with my fixes that you may want to check.

I ran into a quoting issue in how the shlex library handles quotes and what your cmd line program is expecting. And, since I used it on windows, in a posix/non-posix mode problem.

answered May 22, 2012 at 17:48

Spencer Rathbun

15k6 gold badges56 silver badges74 bronze badges

Collectives™ on Stack Overflow

python - using multiple processes / parallel processing

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related