2

I have several fastq.gz files, both R1 and R2 in a directory dir on a Linux system. It looks like:

dir
 |____sampleA_1.fastq.gz
 |____sampleA_2.fastq.gz
 |____sampleB_1.fastq.gz
 |____sampleB_2.fastq.gz
 |____sampleC_1.fastq.gz
 |____sampleC_2.fastq.gz

I wanted to create a txt file with sample name as first column, path to R1 fastq as second column and path to R2 fastq as third column.

Inside dir I tried in the following way:

find "$PWD" -name \*1.fastq.gz > list1.txt
find "$PWD" -name \*2.fastq.gz > list2.txt

And again I have to merge those two files and give a column name and again create another column with sample names. Instead, Is there a way to make the file with a single command?

And txt file should look like below:

sample            Second                    Third
sampleA    dir/sampleA_1.fastq.gz     dir/sampleA_2.fastq.gz
sampleB    dir/sampleB_1.fastq.gz     dir/sampleB_2.fastq.gz
sampleC    dir/sampleC_1.fastq.gz     dir/sampleC_2.fastq.gz
Improve this question
3
  • Is that really the format your files are in? Aren't they something like sampleA_L001_R1_001.fastq.gz and sampleA_L001_R2_001.fastq.gz? Do you really only have one underscore (_) per file name? Also, can you be sure you will never have more than a single pair of reads per sample? Depending on coverage and size of the assay's target regions, you can have several. Commented Jul 18, 2022 at 16:36
  • Assuming there’s a common key between items (sample name?), a clever join might be able to do it Commented Jul 19, 2022 at 3:10
  • You might garner more interest in your question if you add keywords in your first paragraph (something like "Paired-End Sequencing"), and add a reference link. This helps orient potential answerers. Commented Jul 19, 2022 at 15:56

3 Answers 3

Reset to default
3

If you can guarantee that there are always a pair of samples, this bash/ksh code will generate the output based on the presence of all the sample 1 files:

Example (set up demo environment):

mkdir -p /tmp/710303/dir
cd /tmp/710303
touch dir/sample{A,B,C}_{1,2}.fastq.gz       # Assumes a { }-aware shell

File generation (work on the demo environment)

printf "%s %s %s\n" 'sample' 'Second' 'Third'
for f1 in dir/sample*_1.fastq*               # Loop through all first samples
do
    fn="${f1##*/}"; fn="${fn%%_*}"           # Label
    f2="${f1/1/2}"                           # Filename for second sample
    printf "%s %s %s\n" "$fn" "$f1" "$f2"    # Output the values
done

Output

sample Second Third
sampleA dir/sampleA_1.fastq.gz dir/sampleA_2.fastq.gz
sampleB dir/sampleB_1.fastq.gz dir/sampleB_2.fastq.gz
sampleC dir/sampleC_1.fastq.gz dir/sampleC_2.fastq.gz

These are space-separated columns. If you want tab-separated then change the printf format lines to use \t (tab) instead of (space).

Improve this answer
4
  • it should give a path to the second and third columns. No need a path for the first column. And how do I give the names for those columns? Commented Jul 18, 2022 at 16:06
  • @stack_learner titles and path added Commented Jul 18, 2022 at 16:11
  • Thank you very much. In the above script instead of dir I added "$PWD". Commented Jul 18, 2022 at 20:32
  • 1
    The original version of the script would have suited you better, and then you would have just needed to add $PWD to the output values Commented Jul 18, 2022 at 20:34
2

This looks needlessly complicated, but it's handling the case where only one of the sample's files are present

{
    printf '%s\n' sample Second Third

    find ./dir/ -type f -name '*.fastq.gz' -print \
    | cut -d _ -f 1 \
    | sort -u \
    | bash -c '
        while read -r root; do
            echo "${root##*/}"
            for i in 1 2; do
                f="${root}_${i}.fastq.gz"
                [[ -f "$f" ]] && echo "$f" || echo ""
            done
        done
      ' 
} \
| paste - - - \
| column -s $'\t' -t

Testing:

mkdir dir
touch dir/sample{A,B,C}_{1,2}.fastq.gz
touch dir/sample{D_1,E_2}.fastq.gz
touch dir/ignore.me

Then the above command outputs

sample   Second                    Third
sampleA  ./dir/sampleA_1.fastq.gz  ./dir/sampleA_2.fastq.gz
sampleB  ./dir/sampleB_1.fastq.gz  ./dir/sampleB_2.fastq.gz
sampleC  ./dir/sampleC_1.fastq.gz  ./dir/sampleC_2.fastq.gz
sampleD  ./dir/sampleD_1.fastq.gz  
sampleE                            ./dir/sampleE_2.fastq.gz

Maybe this GNU awk version is a bit tidier:

find ./dir -type f | gawk -F/ -v OFS='\t' '
    BEGIN { print "sample", "Second", "Third" }
    match($NF, /^(.*)_([12]).fastq.gz$/, m) {
        file[m[1]][m[2]] = $0
    }
    END {
        PROCINFO["sorted_in"] = "@ind_str_asc"
        for (sample in file)
            print sample, file[sample][1], file[sample][2]
    }
' | column -s $'\t' -t

Produces same output as above.

Improve this answer
2
  • sorry, does your code work if the file are tq.gz instead of fastq.gz? Commented Jul 25, 2022 at 14:56
  • I hardcode "fast.gz" so no. Your question does not require otherwise. Commented Jul 25, 2022 at 18:10
2 
$ cat tst.awk
BEGIN {
    FS="[/_]"; OFS="\t"
    print "sample", "Second", "Third"
}
NR%2 { second = $0; next }
{ print $2, second, $0 }


$ printf '%s\n' dir/* | awk -f tst.awk
sample  Second  Third
sampleA dir/sampleA_1.fastq.gz  dir/sampleA_2.fastq.gz
sampleB dir/sampleB_1.fastq.gz  dir/sampleB_2.fastq.gz
sampleC dir/sampleC_1.fastq.gz  dir/sampleC_2.fastq.gz
Improve this answer
4
  • @Morton instead of dir/* I tries giving "$PWD" but it didn't work. Is there a way for that? Commented Jul 26, 2022 at 14:24
  • $PWD contains the current directory name (Present Working Directory), you need to pass file names to the command. Sounds like you might want ./* or "$PWD"/*. Commented Jul 26, 2022 at 14:37
  • Yes I knew this. I tried "$PWD"/* this, Second and Third columns are ok, but column sample doesn't show the right names, instead it is showing the first directory of the Present Working Directory Commented Jul 26, 2022 at 15:31
  • idk, the script I provided produces the output you asked for from the sample input you provided so if there's some other input it doesn't work for then you'd have to provide that in your question. Looks like you already accepted an answer to this question, though, so you should ask a new followup question. Commented Jul 26, 2022 at 15:37

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.