Bash parallel process on multiple input files, one output file

Question

I created a for loop that goes through multiple files and outputs the results into one file:

for x in /home/moleculo/x*; do ExtractOutCalls2.sh /home/Scripts/000 $x & done

So each of my input files starts with letter x, that's x* as input. Script takes each of those input files $x and outputs to file /home/Scripts/000

Now I have a question:

if this is done on a few thousand files, is it a good way to put like this?

also if I use multiple input files, but specify one output file, will this mean, that my output will won't be appended? If not, how to do it

Regards, Irek

choroba · Accepted Answer · 2013-08-16 14:09:05Z

1

Yes, your output file gets overwritten by each process. Make each script output to its own file, and once all the scripts are finished, concatenate the output:

i=0
for x in /home/moleculo/x* ; do
    ExtractOutCalls2.sh /home/Scripts/000 $x > OUT.$i &
    (( i++ ))
done
wait
cat OUT.* > OUT
rm OUT.*

You have to change the script to output to standard output instead of the file, or make it accept the name of the output file to be created.

answered Aug 16, 2013 at 14:09

choroba

245k27 gold badges221 silver badges304 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Irek Over a year ago

I have the outptu as $argv[1] so this is doing the trick. And if this is done in parallel on couple of thousand of files, how does it look in terms of my cores? Won't it crash?

choroba Over a year ago

@Irek: Do you have thousands of cores? The disk I/O would probably be the bottleneck, though. Do not run thousands of processes.

Irek Over a year ago

Well actually, I do, but I just want to run it smart. But even if I do only 1 file input, will sth like xargs -P 4 -n 1 script.sh speed it? Or is it still no good, since I am going with only 1 file

Ole Tange · Accepted Answer · 2013-08-17 08:33:07Z

1

Often you can use the file - to designate stdout:

for x in /home/moleculo/x*; do ExtractOutCalls2.sh - $x & done

To avoid mixing output use GNU Parallel:

parallel ExtractOutCalls2.sh - {} ::: /home/moleculo/x* > output

answered Aug 17, 2013 at 8:33

Ole Tange

34.1k9 gold badges93 silver badges111 bronze badges

1 Comment

Bruce_Warrior Over a year ago

The second one is better and cleaner.

Collectives™ on Stack Overflow

Bash parallel process on multiple input files, one output file

2 Answers 2

3 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related