Running parallel based on variables from file in bash

Question

I've been trying to run a while-loop in parallel for a work that takes days.

I've seen other answers where parallel was implemented within a while-loop, but for that case it does work in blocks, where the next job only works after all previous jobs finished.

This is the code, which reproduced the two columns of the CSV file:

while IFS="," read fq tab
do
echo $fq
echo $tab
done < /home/samples.csv

where the csv file contain two columns with no header (column 1 and 2), where the variables are stored. For example:

a1,b1
a2,b2
a3,b3
a4,b4

I've been trying to run this in parallel so when a job is finished the other starts immediately because of the long run times.

This is the code:

while IFS="," read fq tab
do
parallel -j 1 --verbose --delay 2  "echo $fq; echo $tab" 
done < /home/samples.csv

But this produces

a1 b1 a1,b1

a1 b1 a2,b2

a1 b1 a3,b3

a1 b1 a4,b4

And not

a1   b1
a2   b2
a3   b3
a4   b4

if you need the results of processing a1 b1 for processing a2 b2 then you can't parallelize the code — Fravadona
– Fravadona, Commented Jan 26, 2022 at 20:13
To get your expected result: with parallel: parallel -j 3 -a /home/sample.csv --colsep ',' echo {1} {2}. Knowing if this will suit your final goal is an other story. — Zeitounator
– Zeitounator, Commented Jan 26, 2022 at 20:21
I don't need the results of a1 and b1 to proceed! They are independent in this case — Gabriel G.
– Gabriel G., Commented Jan 26, 2022 at 20:28
I'm unable to reproduce your ouput (wrong or expected); 'course, it doesn't help that you're trying to parse input lines with IFS="," but there are no commas in the input file; consider reviewing How to create a minimal, reproducible example? and then come back and update your question accordingly, making sure your expected output matches the sample input you provide — markp-fuso
– markp-fuso, Commented Jan 26, 2022 at 20:30

Ole Tange · Accepted Answer · 2022-01-27 12:29:47Z

4

cat <<_EOF > samples.csv
a2,b2
a3,b3
a4,b4
_EOF

cat samples.csv | parallel --colsep , echo column 1 = {1} column 2 = {2}

If samples.csv is TAB separated, ; separated, or separated with spaces:

cat samples.csv | parallel --colsep '\t' echo column 1 = {1} column 2 = {2}
cat samples.csv | parallel --colsep ';' echo column 1 = {1} column 2 = {2}
cat samples.csv | parallel --colsep ' +' echo column 1 = {1} column 2 = {2}

answered Jan 27, 2022 at 12:29

Ole Tange

34.1k9 gold badges93 silver badges111 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Running parallel based on variables from file in bash

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related