0

I have a csv file containing a list of patterns like this

file 1:

aaa;bbb      
ccc;ddd
eee;fff
...

file 2: 

aaa2222222222222222
3333333333333333333
4444444444444444444
bbb555555555555555
8888888888888888888
ccc5555555555555555
5555555555555555555
0000000000000000000
ddd6666666666666666

How can I use the 2 patterns in each row (e.g. aaa & bbb) from file 1 in a sed -n '/aaa/,/bbb/p' command to match and print lines in between the matching patterns from file 2, save the output and then replace the command with the next pair of patterns to repeat the process?

The output I´m expecting is 

333333333333333         
444444444444444

saved to a new file as they are the lines between aaa & bbb

Then use a loop to find the next set of lines between ccc & ddd, etc.

Improve this question
6
  • That file you provided is the list of patterns, can you provide a sample of the file that you run the sed command on? Commented May 31, 2020 at 14:43
  • 2
    Better add expected output Commented May 31, 2020 at 14:44
  • Hi yes awk is also fine. Basically aaa and bbb are two strings that I want to find in a different file, and then extract the lines in between those two matching strings from that file. I have a csv file with a list of these strings that I want to look up. And instead of entering the terms one by one ito a sed command to extract the lines in between, I wanted to know if it is possible to use a loop to print lines inbetween these multiple patterns. Commented May 31, 2020 at 14:47
  • 1
    Do you want to find aaa, etc. anywhere on the line or only at the start of the line? Do you want to do a regexp or string comparison? Can you have overlapping ranges in files1? How about nested ranges? How about duplicate ranges or duplicate start or end values? Commented May 31, 2020 at 20:08
  • basically patterns like aaa are tags and teach tag occurs only once in the file. I just want to extract the lines in between the 2 lines containing these tags. Commented Jun 1, 2020 at 8:36

5 Answers 5

Reset to default
1

This may be what you're looking for, assuming no overlapping or nested ranges or duplicate begin/end strings specified in file1 or present in file2:

$ cat tst.awk
BEGIN { FS=";" }

NR==FNR {
    begs2ends[$1] = $2
    next
}

end == "" {
    for ( beg in begs2ends ) {
        if ( index($0,beg) == 1 ) {
            end = begs2ends[beg]
            close(out)
            out = "out" (++cnt) ".txt"
            break
        }
    }
    next
}

{
    if ( index($0,end) == 1 ) {
        end = ""
    }
    else {
        print $0 " > " out
    }
}

.

$ awk -f tst.awk file1 file2
3333333333333333333 > out1.txt
4444444444444444444 > out1.txt
5555555555555555555 > out2.txt
0000000000000000000 > out2.txt

Change print $0 " > " out to print > out to actually create the separate output files when you're done with initial testing.

Improve this answer
5
  • Ed, this does not treat the file aaa2|3333|4444|bbb5|8888|ccc5|0000|aaa7|7777|bbb7|ddd6| correctly (substitute pipes by newlines). out1.txt will not contain 7777. Commented May 31, 2020 at 21:12
  • 1
    @Quasímodo I updated my answer to make it clearer it assumes no duplicate begin or end strings, I haven't heard back from the OP about that or the other non-trivial possible cases yet (see unix.stackexchange.com/questions/590106/…). Commented May 31, 2020 at 22:00
  • 1
    My fault, Ed, I missed that information on the top of your answer. Commented May 31, 2020 at 22:11
  • 1
    @Quasímodo No problem, looks like the OP doesn't have any duplicate tags, see unix.stackexchange.com/questions/590106/…. Still several important unanswered questions though! Commented Jun 1, 2020 at 13:43
  • 1
    Thank you all your codes worked. My patterns file was corrupted. That seemed to be the reason why couldn´t extract only the necessary lines initially. Thank you Commented Jun 1, 2020 at 14:01
1

I favor awk because this way only a single process is needed. Going with sed would probably require inefficient shell loops and multiple sed commands.

awk -F ';' '
    NR==FNR{        #For the first file given as an argument
        k++         #Increment k
        pat1[k]=$1  #Add 1st field (before ;) to pat1 array
        pat2[k]=$2  #Add 2nd field (after ;) to pat2 array
        next        #Start new cycle with the next line
    }

    {
        for (i=1;i<=k;i++){
            if ($0~pat2[i]){flag[i]=0}           #If line has terminating pattern
            if (flag[i]){print>(pat1[i]pat2[i])} #If flag=1, print this line to a file
            if ($0~pat1[i]){flag[i]=1}           #If line has starting pattern
        }
    }
' patterns.csv input

Care not to invert arguments: First comes the patterns file, then comes the input file.

Sample input: 

aaa2
3333
4444
bbb5
8888
ccc5
0000
aaa7
7777
bbb7
ddd6

Two files are created, with contents:

$ cat aaabbb
3333
4444
7777
$ cat cccddd
0000
aaa7
7777
bbb7
Improve this answer
6
  • Thank you for the code, but my output is like this cat 'cccddd'$\r' 5555555555555555555 0000000000000000000 ddd6666666666666666 $ cat 'aaabbb'$'\r' 3333333333333333333 4444444444444444444 bbb555555555555555 8888888888888888888 ccc5555555555555555 5555555555555555555 0000000000000000000 ddd6666666666666666 I think it prints everything from the first field term to the end of the file Commented May 31, 2020 at 16:39
  • That \r is suspicious, looks like you have carriage-returns... Do your files come from Windows? Run dos2unix in your files and try again. I've tested it on the file you provided and it worked correctly. Commented May 31, 2020 at 17:11
  • 1
    Being in Linux does not guarantee you do not have carriage-return in your file. Do you get any output if you run od -c file | grep '\\r' in any of the files? Commented May 31, 2020 at 18:21
  • 1
    print>pat1[i]pat2[i] is undefined behavior per POSIX, it needs to be print>(pat1[i]pat2[i]) to be portable to all awks. With any awk but GNU awk it'll fail with a "too many open files" error once you've created a dozen or so output files though, you need to close() the output files as you go if you want it to be portable to non-GNU awks. Commented May 31, 2020 at 20:06
  • 1
    @EdMorton Thank you for your observation. I've improved the first point you mentioned, but I don't think there is any way to handle the second point because repeated open and closes will affect performance. I will assume that either the patterns are not too many or that OP has a capable awk. Commented May 31, 2020 at 21:12
1

with Gnu sed we prepare the sed code dynamically from the contents of file1 and then apply the generated code to file2. The files are created silently. 

sed -Ee 's|;|/,/|' \
     -e 's|(.).*(.)|/&/{\n\t//!w FILE_\1\2\n}|' file1 \
|  sed -nf - file2
more FILE_??
Improve this answer
1
  • wooooooww this is some darky nasty pretty little piece of code! I never used addresses in sed before loved it dude! Commented Jun 1, 2020 at 12:55
0

Tried to make all with awk but I couldnt, anyway...you can use awk to phrase and print de code that sh will run and use the sed code you posted.

pattern_file="file1"
data_file="file2"
awk -v data_file=$data_file -F";" '{print "sed -n \047/" $1 "/,/" $2 "/p\047 " data_file " | sed \047/[" $1 "|" $2 "]/d\047 > " $1$2 ".txt"}' $pattern_file

The command send to the shell is:

sed -n '/aaa/,/bbb/p' file2 | sed '/[aaa|bbb]/d' > aaabbb.txt
sed -n '/ccc/,/ddd/p' file2 | sed '/[ccc|ddd]/d' > cccddd.txt
sed -n '/eee/,/fff/p' file2 | sed '/[eee|fff]/d' > eeefff.txt

The output will be saved to aaabbb.txt and so on.

Improve this answer
0 
file2content="$( < file2 )";
while read p;do 
    ((${#p}))&&sed -n "/${p%;*}/,/${p##*;}/p" <<< "${file2content}"|grep -v "^${p%;*}\|^${p##*;}" > "${p%;*}_${p##*;}"; 
done <file1
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.