running a command in shell script in parallel without using background

Question

This is my current script which runs another script on multiple files (filename0, filename1, etc). My problem is that it takes time to run and so would like to run them in parallel. Most solutions involve backgrounding the jobs. However if I do this then the job just stops. The way I get round this at the moment is to run this command in different terminal windows in batches of 10.

for i in `seq 0 100` 
do
my-other-script filename$i
done

Please don't wield the ancient seq command. It is older than time. Use for i in {0..100} to iterate over numbers. And as a constant reminder: always quote parameter expansions. Using "filename$i" would be safer and you wouldn't have to worry about filenames containing whitespace. — user1019830
– user1019830, Commented Jun 6, 2013 at 11:41

l0b0 · Accepted Answer · 2013-06-05 09:59:51Z

2

Jobs don't "just stop" because they're backgrounded, but you have to wait for them to finish to avoid exiting the script before they're done. Documentation

answered Jun 5, 2013 at 9:59

l0b0

59.6k32 gold badges155 silver badges247 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

user1958508 Over a year ago

If I do a CTRL-Z to suspend the job it says 'stopped' instead of 'Suspended' which is the problem I am having

l0b0 Over a year ago

Please read the documentation. If you stop it you can either fg it or bg it, both leading to continuing execution.

user1958508 Over a year ago

I cannot bg the job. If I do 'my-script filename1 &' I do not get a command line back, the job stops completely. It disappears from my 'jobs' list.

William Pursell Over a year ago

@user1958508 Your job is stopping because it is done. All of its children are happily running in the background, but they are not children of your current shell so they don't appear in the jobs list. Your script should spawn its children in the background and then wait for them to finish.

Ole Tange · Accepted Answer · 2013-06-06 11:29:21Z

0

GNU Parallel is made for precisely these kind of tasks:

seq 0 100 | parallel -j10 my-other-script filename{}

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

wget -O - pi.dk/3 | sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

answered Jun 6, 2013 at 11:29

Ole Tange

34.1k9 gold badges93 silver badges111 bronze badges

Collectives™ on Stack Overflow

running a command in shell script in parallel without using background

2 Answers 2

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related