Shell script: Parallelizing commands in bash script under Ubuntu linux

Question

I am creating a bash script to losslessly compress jpeg and png files, so I use two package under Ubuntu named jpegoptim and optipng. A will use the script with 2 or 4 core processors. I would like to use the full capacity of my CPU.

The problem is that optipng does not support multithreading by default (it only uses one CPU core), so I decided, I launch 2 or 4 parallel processes to compress the images faster. I already sorted the image files to 4 almost equal arrays (based on number of pixels), and now I need to run the processes parallelized.

I am trying parallelizing the processes with the & character at the end of the command, but it does not do the job parallel.

optipngout=$(optipng -$pnglevel -dir $outdir ${threaddata_1[@]} &)
optipngout=$(optipng -$pnglevel -dir $outdir ${threaddata_2[@]} &)
optipngout=$(optipng -$pnglevel -dir $outdir ${threaddata_3[@]} &)
optipngout=$(optipng -$pnglevel -dir $outdir ${threaddata_4[@]} &)

I have to catch the output of the command, so I think the problem is with the $() structure.

I don't think bash has concurrency support, but I could be mistaken. Any reason you can't use a full programming language? — beatgammit
– beatgammit, Commented Mar 22, 2013 at 9:47
The util taskset is something you might look into. Although I must say, I haven't been able to realize much benefit if any in my limited use of it. Also, the term "processor affinity" might help you in your searches. — jedwards
– jedwards, Commented Mar 22, 2013 at 9:49
@jedwards it is working very well with the & character. Thanks for your help! — Patartics Milán
– Patartics Milán, Commented Mar 22, 2013 at 11:37

Lars Kotthoff · Accepted Answer · 2013-03-22 10:08:34Z

3

Have a look at Gnu parallel. It allows you to do what you want with only small modifications to your existing script.

answered Mar 22, 2013 at 10:08

Lars Kotthoff

109k16 gold badges210 silver badges210 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2017-05-23 12:29:53Z

2

I solved my problem with the taskset and & commands. Thanks for @jedwards

The final code looks like this:

outstr=$(taskset 0xFFFFFFFF optipng -$pnglevel -dir $outdir ${threaddata_1[@]}) &
outstr=$(taskset 0xFFFFFFFF optipng -$pnglevel -dir $outdir ${threaddata_2[@]}) &
outstr=$(taskset 0xFFFFFFFF optipng -$pnglevel -dir $outdir ${threaddata_3[@]}) &
outstr=$(taskset 0xFFFFFFFF optipng -$pnglevel -dir $outdir ${threaddata_4[@]}) &

edited May 23, 2017 at 12:29

CommunityBot

11 silver badge

answered Mar 22, 2013 at 12:00

Patartics Milán

4,9784 gold badges29 silver badges32 bronze badges

Collectives™ on Stack Overflow

Shell script: Parallelizing commands in bash script under Ubuntu linux

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related