Using a for loop along with multi threading python

Question

I have a parent script that runs 10 child scripts. These scripts are identical apart from the 'name', each script calls a different api, saves to a different folder etc that depends on name.

I originally just copied the script 10 times however I now realise this is ridiculous as if I make one upgrade I have to do it to all 10.

I'm quite new to threading and am not sure if this is allowed, but can I set up a for loop or similar that calls the script and places the 'name' inside:

Ie

#parent.py
from threading import Thread
import sys
names =  ['BTC', 'BTS', 'ETH', 'CLAM', 'DOGE', 'FCT', 'MAID', 'STR', 'XMR', 'XRP' ]

for name in names: 
    sys.path.append('/python/loanrates/'+name)
    import name
    Thread(target=name.main(name)).start()
    Thread(target=name.main(name)).join()

Yes you can, but you shouldn't include the join in the loop, as it waits for your thread to finish and will block the start of other threads — CoMartel
– CoMartel, Commented Feb 22, 2016 at 14:48

ᴀʀᴍᴀɴ · Accepted Answer · 2016-02-22 14:47:41Z

1

You can have list of threads , start them in a loop and then wait for them to join in another for loop:

from threading import Thread
import sys
names =  ['BTC', 'BTS', 'ETH', 'CLAM', 'DOGE', 'FCT', 'MAID', 'STR', 'XMR', 'XRP' ]
threads = []
for name in names: 
    sys.path.append('/python/loanrates/'+name)
    import name
    T = Thread(target=name.main(name))
    threads.append(T)

for thread_ in threads:
    thread_.start()

for thread_in threads:
    thread_.join()

answered Feb 22, 2016 at 14:47

ᴀʀᴍᴀɴ

4,5368 gold badges41 silver badges61 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ami Tavory · Accepted Answer · 2016-02-22 14:53:33Z

1

To begin with, you should consider using the multiprocessing module, as threading is misleading in Python (in some sense, Python is currently single threaded).

Using this module, it's very easy to do "push the loop" on to the module.

First, define a pool:

# start 4 worker processes
pool = Pool(processes=4)

Now, if you have a function f that runs on a name, you can use

pool.map(f, names)

This will return only when f was applied to each item in names, using the Pool of 4 processes (in this case). I.e., it's like a "magical" parallel for loop

"parallel for" name in names:
    f(name)

edited Feb 22, 2016 at 14:53

answered Feb 22, 2016 at 14:47

Ami Tavory

76.7k13 gold badges152 silver badges196 bronze badges

2 Comments

David Hancock Over a year ago

This is interesting, thanks for the insight, however i just learnt how to use threading. Don't have the time to en devour down this road for now

Ami Tavory Over a year ago

It's basically the same for the threading module, though. Just beware that you won't get any real parallelism from it.

Collectives™ on Stack Overflow

Using a for loop along with multi threading python

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related