Getting data from matching list

Question

list.txt

 GETID_17049_knownids_1/2_Confidence_0.625_Length_2532
 GETID_9248_knownids_6/10_Confidence_0.439_Length_2474
 GETID_11084_knownids_3/3_Confidence_0.600_Length_1451
 GETID_15916_knownids_10/11_Confidence_0.324_Length_1825

sample1.txt

>GETID_17049_knownids_1/2_Confidence_0.625_Length_2532
sampletextforsample1
sampletextforsample1
sampletextforsample1
>GETID_18457_knownids_1/2_Confidence_0.625_Length_2532
sample2textforsample1
sample2textforsample1
sample2textforsample1
sample2textforsample1

sample2.txt

>GETID_11084_knownids_3/3_Confidence_0.600_Length_1451
sampletextforsample2
sampletextforsample2
>GETID_67838_knownids_3/3_Confidence_0.600_Length_1451
sample2textforsample2
sample2textforsample2

sample3.txt

>GETID_17049_knownids_1/2_Confidence_0.625_Length_2532
sampletextforsample3
sampletextforsample3
sampletextforsample3
>GETID_15916_knownids_10/11_Confidence_0.324_Length_1825
sample2textforsample3
sample2textforsample3

output.txt

>GETID_17049_knownids_1/2_Confidence_0.625_Length_2532
sampletextforsample1
sampletextforsample1
sampletextforsample1
>GETID_17049_knownids_1/2_Confidence_0.625_Length_2532
sampletextforsample3
sampletextforsample3
sampletextforsample3
>GETID_11084_knownids_3/3_Confidence_0.600_Length_1451
sampletextforsample2
sampletextforsample2
>GETID_15916_knownids_10/11_Confidence_0.324_Length_1825
sample2textforsample3
sample2textforsample3

I want to read each line from list.txt (by capturing bracketed values (GETID_{17049}knownids{1/2}_Confidence_1.0_Length_{2532}) and compare with sample1.txt, sample2.txt, sample3.txt has multiple line and print the contents (output.txt) from these files when it is matched with list.txt. The output should contain exact match to list.txt. Any help in awk/sed/perl is appreciated.

Just curious, how close does a grep -F -A 1 -f list.txt sample[123].txt | grep -vx '--' get to what you want? — jw013
– jw013, Commented Aug 29, 2012 at 18:43
I tried grep and awk script from this previous post.But it does not work because I have to compare with three sample files — jack
– jack, Commented Aug 29, 2012 at 18:58

Community · Accepted Answer · 2017-04-13 12:36:41Z

1

If you slightly modify the solution provided by Gilles in this question (also referred to by jw013), you can get the effect you ask for, except that the order is based on the input sequence and not identical to output.txt listed in your question:

awk -v patterns_file=list.txt '
BEGIN {
  while (getline < patterns_file)
    patterns_array[">" $0] = 1
  close(patterns_file)
}
$0 in patterns_array { print; getline; print }
' sample[1-3].txt

Output:

>GETID_17049_knownids_1/2_Confidence_0.625_Length_2532
sampletextforsample1
>GETID_11084_knownids_3/3_Confidence_0.600_Length_1451
sampletextforsample2
>GETID_17049_knownids_1/2_Confidence_0.625_Length_2532
sampletextforsample3
>GETID_15916_knownids_10/11_Confidence_0.324_Length_1825
sample2textforsample3

Edit

To make this work with multi-line records use an appropriate record separator (RS), in your case setting it to: greater-than at beginning of file (^>) or new-line followed by greater-than (\n>) or new-line at the end of the file (\n$) seem like good choices based on the input provided.

Something like this should work:

awk -v patterns_file=patterns.txt '
BEGIN {
  while (getline < patterns_file) 
    patterns_array[$0] = 1
  close(patterns_file)
  RS="^>|\n>|\n$"
}
$1 in patterns_array { print ">" $0 }
' sample[1-3].txt

Edit 2

To output each record only once, delete it from patterns_array after output:

awk -v patterns_file=patterns.txt '
BEGIN {
  while (getline < patterns_file) 
    patterns_array[$0] = 1
  close(patterns_file)
  RS="^>|\n>|\n$"
}
$1 in patterns_array { print ">" $0; delete patterns_array[$1] }
' sample[1-3].txt

edited Apr 13, 2017 at 12:36

CommunityBot

1

answered Aug 30, 2012 at 14:04

Thor

17.5k3 gold badges55 silver badges71 bronze badges

See edit to the answer.

Thor
– Thor

2012-08-30 15:08:10 +00:00
Commented Aug 30, 2012 at 15:08
now it works.The ouput prints without '>' symbol. Also the output should contain exact count as in the list.txt.

jack
– jack

2012-08-30 15:42:45 +00:00
Commented Aug 30, 2012 at 15:42
Forgot to re-add >, fixed. What do you mean by exact count?

Thor
– Thor

2012-08-30 15:55:08 +00:00
Commented Aug 30, 2012 at 15:55
say if count for '>' in the list.txt, the output should also contains same count for '>', even there is repeat in list or sample

jack
– jack

2012-08-30 16:06:17 +00:00
Commented Aug 30, 2012 at 16:06
Just delete them from patterns_array after output. I'll add an edit.

Thor
– Thor

2012-08-30 16:17:55 +00:00
Commented Aug 30, 2012 at 16:17

Add a comment |

terdon · Accepted Answer · 2012-08-30 15:20:57Z

Here's a PERL solution. It will work for any number of files and expects the first file to be the list. It will also append the file name to the FASTA header.

#!/usr/bin/perl -w
use strict;
my $list=shift;
open(A,$list); 
my %k;
while(<A>){
    ## Remove trailing newline
    chomp;
    if ( /(\d+?)_knownids_(.+?)_.+?(\d+)$/){ 
      ## Concatenate the patterns and save in a hash
      my $pp=join("-", $1,$2,$3);
      $k{PAT}{$pp}=$_;
    }
}
close(A);
## Read each input file
my $name;
for my $f (@ARGV) {
    open(F,$f);
    while(<F>){
       ## Skip empty lines
       next if /^\s*$/;
       ## Is this a FASTA header?
       if ( /^\s*>/){
           ## If this id is in the list, keep it for this file
           if(/(\d+?)_knownids_(.+?)_.+?(\d+)$/){ 
              $name=join("-", $1,$2,$3);
           }
           ## Skip the sequences we are not interested in
           else{$name="foo"}
       }
       ## Collect the sequence
       else {
           if (defined($k{PAT}{$name})) {
           $k{$f}{$name}.=$_;
           }   
       } 
    }
    close(F);
}
## For each unique pattern found in list.txt
foreach my $pat (keys(%{$k{PAT}})) {
    ## For each of the files passed as arguments
    foreach my $file (@ARGV) {
    ## If the pattern was found in that file, print
    if (defined($k{$file}{$pat})) {
          print ">$k{PAT}{$pat}_$file\n";  
          print "$k{$file}{$pat}"
        }
    }
}

If the script is saved as compare.pl, do:

$ ./compare.pl list.txt sample1.txt sample2.txt sample3.txt sampleN.txt

The output is:

> GETID_11084_knownids_3/3_Confidence_0.600_Length_1451_sample2.txt
sampletextforsample2
> GETID_17049_knownids_1/2_Confidence_0.625_Length_2532_sample1.txt
sampletextforsample1
> GETID_17049_knownids_1/2_Confidence_0.625_Length_2532_sample3.txt
sampletextforsample3
> GETID_15916_knownids_10/11_Confidence_0.324_Length_1825_sample3.txt
sample2textforsample3

there is some warning:Use of uninitialized value in join or string at line 26. — jack
– jack, Commented Aug 30, 2012 at 15:02
@jack, that warning probably means you have a FASTA header that does not contain "knownids". I have edited the script to avoid these errors. Make sure ALL your FASTA headers conform to the pattern you gave in you question. — terdon
– terdon ♦, Commented Aug 30, 2012 at 15:23
Thanks.now it works.I need exact counts as in the list.txt even samples contains identical ids. — jack
– jack, Commented Aug 30, 2012 at 15:41

Stack Exchange Network

Getting data from matching list

2 Answers 2

Edit

Edit 2

You must log in to answer this question.

Linked

Hot Network Questions

Getting data from matching list

2 Answers 2

Edit

Edit 2

You must log in to answer this question.

Linked

Related

Hot Network Questions