Getting syntax error using awk in parallel processing

Question

I have 44 .tsv files in one folder and I want to calculate the number of intersect of each pairwise with intersect command of bedtools tool. each output file would have 4 columns and I just need to save only sum of value of column 4 in each output file. I can do it easily when I do it by one one but when I use parallel processing to do the whole process at the same time I get syntax error

Here is the code and result when I try each two pairs by one one manually

$ bedtools intersect -a p1.tsv -b p2.tsv -c

chr1    1   5   1

chr1    8   12  1

chr1    18  20  1

chr1    21  25  0

bedtools intersect -a p1.tsv -b p2.tsv -c | awk '{sum+=$4} END {print sum}

3

Here is the code and result when I am using parallel processing

$ parallel "bedtools intersect -a {1} -b {2} -c |awk '{sum+=$4} END {print sum}'> {1}.{2}.intersect" ::: `ls *.tsv` ::: `ls *.tsv`

awk: cmd. line:1:{sum+=} END {print sum}
awk: cmd. line:1:            ^ syntax error
awk: cmd. line:1:{sum+=} END {print sum}
awk: cmd. line:1:            ^ syntax error
awk: cmd. line:1:{sum+=} END {print sum}
awk: cmd. line:1:            ^ syntax error
awk: cmd. line:1:{sum+=} END {print sum}
awk: cmd. line:1:            ^ syntax error

The result should be 44*44 files that contain one single value foe example just 3

Ole Tange · Accepted Answer · 2019-07-02 15:27:29Z

4

@DudiBoy has a good solution. But to me it is annoying that I have to make another file just because I want to call GNU Parallel.

So you can also use functions. This way you do not need to make a new file:

doit() {
  bedtools intersect -a "$1" -b "$2" -c | awk '{sum+=$4} END {print sum}'
}
export -f doit

parallel --results {1}.{2}.intersect doit {1} {2} ::: *.tsv ::: *.tsv

answered Jul 2, 2019 at 15:27

Ole Tange

34.1k9 gold badges93 silver badges111 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Mark Setchell · Accepted Answer · 2019-07-02 10:42:45Z

2

I think you need to quote it like this:

parallel bedtools intersect -a {1} -b {2} -c \| awk \'{sum+=\$4} END{print sum+0}\' \> {1}.{2}.intersect ::: *tsv ::: *tsv

answered Jul 2, 2019 at 10:42

Mark Setchell

210k32 gold badges309 silver badges503 bronze badges

Comments

Dudi Boy · Accepted Answer · 2019-07-02 13:13:11Z

2

I believe @MarkSetchell is valid answer. You can also try to clean it up by inserting your complicated line into a bash script you can test.

intersect.bash

 #!/bin/bash
 bedtools intersect -a $1 -b $2 -c | awk '{sum+=$4} END {print sum}'

Test intersect.bash to function correctly, then parallel it.

parallel intersect.bash {1} {2}

good luck.

answered Jul 2, 2019 at 13:13

Dudi Boy

1

Collectives™ on Stack Overflow

Getting syntax error using awk in parallel processing

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related