1

I have two files:

Ref.txt which contains:

ABCDEFG,15147,ABC,ABCD,ABCDE
HIJKLMN,500,HIJKLM,HIJKL,HIJK

and Seq.txt which contains:

ABCDEFG
ABCXXXX
ABCDXXX
ABCDEXX
HIJKXXX
HIJKLXX
HIJKLMX
HIJKLMN

I would like to search for fields 3-end of each line of the Ref.txt file in the Seq.txt file and replace the lines where that string is found with the string in the first field of that line. The output I am lookin for in this example is this:

ABCDEFG
ABCDEFG
ABCDEFG
ABCDEFG
HIJKLMN
HIJKLMN
HIJKLMN
HIJKLMN

Although this is incorrect code I am thinking of a command like this:

awk '{for(i=3; i<=NF; i++) gsub( $i , $1)}

but where the for loop refers to the the Ref.txt file and the gsub command is executed on the Seq.txt file.

Improve this question

3 Answers 3

Reset to default
2

One way you might consider doing this is to construct a single regular expression from fields 3-end of each line in Ref.txt by chaining them with |, saving that in associative array keyed on the $1 value.

You can then loop over the keys for each $1 of Seq.txt:

awk -F, '
  NR==FNR {
    # construct a single ERE as $3|$4|$5|etc. keyed on $1
    for(i=3;i<=NF;i++) r[$1] = r[$1] == "" ? $i : r[$1] "|" $i;
    next
  } 
  {
    # test $1 against each ERE and substitute the first matching key
    for(k in r) {
      if ($1 ~ r[k]) {
        $1 = k; 
        break
      }
    }
  }
  1
' Ref.txt Seq.txt
Improve this answer
4
  • this seems to be working great so far! Thank you! I think I generally understand how this is working, but I am very new to coding, if you have time I would appreciate a detailed explanation of what is being done in each part of this script so I can modify it myself in the future if I encounter any problems. Commented Aug 14, 2019 at 12:07
  • How would you change the second half of the script if instead of the array being filled with sequences to search for it is filled with line numbers, and I would line to change those lines to the content of the first field of the Ref.txt file? Basically get the same output but using line numbers instead of searching for a sequence. Commented Aug 16, 2019 at 16:40
  • 1
    @DNA_bash so $3 to $NF would be a list of line numbers, rather than a list of patterns? if so, the above approach would not really be suitable Commented Aug 16, 2019 at 17:31
  • that is correct, since a different approach is needed for that problem I have posted a new question to deal with using line numbers as an input here: unix.stackexchange.com/questions/535927/… Commented Aug 16, 2019 at 18:49
0

You can accomplish this with sed using just the Posix constructs as shown here:

$ sed -ne '
   /,/!G
   y/\n_/_\n/
   s/^\([^_][^_]*\).*__\(.*_\)\{0,1\}\([^,]*\),[^,]*,[^_]*,\1,.*/\3/p
   s/^[^_]*__//;s/$/,/
   y/_\n/\n_/
   /\n/!H
' Ref.txt Seq.txt

Note: this is run on a bash command line and the order and number of arguments is as given here.

Perl can also be used here without taking recourse of the regexes:

$ perl -F, -lane '
   if ( @ARGV ) {
     $h{$_} = $F[0] for @F[2..$#F];
     next;
   }
   my $seq = $_;
   my($k) = grep { ! index($seq, $_) } keys %h;
   print $h{$k};
' Ref.txt Seq.txt
Improve this answer
-1

If you are lazy like me, you don't want to deal with arrays and loops and leave that up to your tool. That's why I prefer sed and just throw those Ref.txt lines (identified by the comma) on the hold space with H. Actually I add another comma, so I know that patterns are always in between two commas. So together it's /,/{s/$/,/;H;d;}.

Now, for each line of Seq.txt, I append the hold space with my reference pile with G and I let the s command replace the pattern with the pattern found in the pile. As often, the regular expression is easier to write than to read.

Let's look at the pattern space now:

ABCDEFG\n\nABCDEFG,15147,ABC,ABCD,ABCDE\nHIJKLMN,500,HIJKLM,HIJKL,HIJK
\_____/    \_____/       \_/
replace    by this  if  match
  • The part to be replaced is at the beginning (^) with a matching part ([A-Z]+) and some parts before and after ([A-Z]*)
  • The substitute is a sequence between a newline and a comma: \n[A-Z]*,
  • the match is a pattern between two comma, backreferenced from the first part: ,\1,

All together this gives

sed -E '/,/{s/$/,/;H;d;};G;s/^[A-Z]*([A-Z]{1,})[A-Z]*\n.*\n([A-Z]*),[A-Z0-9,]*,\1,.*/\2/;P;d' Ref.txt Seq.txt
Improve this answer
2
  • hmm, this solution doesn't appear to work in my hands. I am very new to coding in general so could be my implementation, but running this outputs a copy of the Seq.txt file with no changes Commented Aug 14, 2019 at 12:04
  • I only could test it with GNU sed, and I see it doesn't work with option --posix. Currently I don't see why, but replacing the [A-Z]+ with [A-Z]{1,} seems to help, although those should behave the same. Sorry for that. Edited the answer. Commented Aug 14, 2019 at 12:46

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.