Threading subprocesses in Python

Question

If I use Python's Threading library can I accomplish a batch of subprocess processes faster? Let's say for instance I need to convert 100 .wav files to .mp3 files. If I wrap 'ffmpeg' in a Python script that used threading could I accomplish the task much faster? Does threading enable me to actually use all 8 threads available in my i7?

I recently stumbled across a Python script that pings a list of hosts but utilizes threading to make things faster. The script seemed succinct and was easy enough for a beginner such as myself to read and understand. So this is where my other question arises: What is all the talk about running parallel tasks being so complex? Maybe I don't understand parallel and parallel != threading. If it is that simple then why wouldn't people use threading for any and all batch conversions when running on a modern processor?

dolphin · Accepted Answer · 2014-06-08 13:01:37Z

4

Here is a simple and working solution for your case. I use it often and it proves to be very useful! This code creates as many threads as cores and lets them execute a (large) number of tasks (in this case, calling a shell program):

import Queue
import threading
import multiprocessing
import subprocess

q = Queue.Queue()
for i in range(30): #put 30 tasks in the queue
    q.put(i)

def worker():
    while True:
        item = q.get()
        #execute a task: call a shell program and wait until it completes
        subprocess.call("echo "+str(item), shell=True) 
        q.task_done()

cpus=multiprocessing.cpu_count() #detect number of cores
print("Creating %d threads" % cpus)
for i in range(cpus):
     t = threading.Thread(target=worker)
     t.daemon = True
     t.start()

q.join() #block until all tasks are done

answered Jun 8, 2014 at 13:01

dolphin

1,26313 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

promaxdev Over a year ago

just curious, how would the worker end, as the while loop exit condition is not apparent. Also, can something be added to 'q' using q.put() after q.join().

BrainStorm · Accepted Answer · 2011-08-02 03:34:58Z

2

making your program multithreaded with threading will certainly make it faster if used correctly, but you can't really control how much of your processor, or how many cores your app will use because it's all handled by the OS, the multiprocessing module in python is another approach you could take, but it would be better if you read about this subject with greater detail before trying out these modules, I recommend this tutorial.

answered Aug 2, 2011 at 3:34

BrainStorm

2,0461 gold badge16 silver badges23 bronze badges

2 Comments

user624208 Over a year ago

Awesome! Thanks for the input I'll check out the tutorial.

BrainStorm Over a year ago

Vote up would be appreciated. =P

Kracekumar · Accepted Answer · 2011-08-02 03:28:59Z

0

You should use multiprocessing module in python, because you can make use more than one core in your computer.Try this multiprocess module in python

answered Aug 2, 2011 at 3:28

Kracekumar

20.5k11 gold badges50 silver badges56 bronze badges

Collectives™ on Stack Overflow

Threading subprocesses in Python

3 Answers 3

1 Comment

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related