traverse array sent from bash, in external awk script

Question

In my bash script I have:

patterns=( "somee? +pattern" "Another +here" )

awk -v l="$line" -f horsepower <(printf '%s\n' "${patterns[@]}") <<<"$stdout"

, so sending to awk script two files, first being process substitution output of printf on ${patterns[@]} array, and second being HERESTRING feed from $stdout variable.

, then in that external awk script, named here horsepower I have the following:

BEGIN {
    ...some initialization, not of importance for this question
}


# while reading first file, populate `patterns` array in awk
FNR==NR { patterns[$0]=""; next }

for (pat in patterns)
    # match lines from `$stdout` on patterns sent from bash, and save them in `a_rem`
    # array, for later output in END block.
    $0 ~ pat { a_rem[NR]=$0 }

# then follows END block
END {
    ...also non important
}

, but I am given notice about syntax error on for from above for (pat in patterns).

What am I doing wrong? Also, is awk getting correct first, second files from my bash script, using that method of <(), and <<<"$stdout"?

I have to mention, all of this was working when I was using literal regexes in awk script, but the moment I introduced that for (pat in patterns), and adding that <() in bash(<<<"$stdout" was ok!), I am getting this syntax error.

Jonathan Leffler · Accepted Answer · 2014-07-10 23:25:20Z

3

There are at least two problems, one at shell level, one at the awk level.

Shell

A program like awk won't read standard input if you give it another file to read. The process substitution <(...) gives it one file to read. To get it to read standard input too, you need:

awk -f program <(...) - <<<"$stdout"

Awk

The pattern/action language applies at the source level. You're trying to apply it dynamically to the list of patterns you read. You'll need to modify the code that reads:

for (pat in patterns)
    # match lines  on patterns sent from bash, and save them in `a_rem`
    # array, for later output in END block.
    $0 ~ pat { a_rem[NR]=$0 }

so that it is more like:

{
    for (pat in patterns)
        if ($0 ~ pat) { a_rem[NR]=$0; break }
}

The added break is an optimization so that you don't add a single line multiple times to the same element of the array.

answered Jul 10, 2014 at 23:25

Jonathan Leffler

759k145 gold badges961 silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

branquito Over a year ago

What should I search for, to learn more about this (dynamically applying to the list of patterns)? And question about break, is it for not to repeat same line, only for the time being in that very same line?

Jonathan Leffler Over a year ago

There's nothing much wrong with what you did that 10 years of experience won't fix. I learned much of this stuff so long ago that the way I learned is simply not relevant today (hint: there wasn't a WWW to search back then). That said, for awk, the GNU gawk manual is pretty good. There is an O'Reilly book on awk and sed; there is also a book by Messrs Aho, Weinberger and Kernighan, whose initials you might recognize.

Jonathan Leffler Over a year ago

Regarding break: suppose there are ten patterns, and a particular line matches five of those patterns. Then the code without the break will store $0 into a_rem[NR] 5 times, whereas the code with break will store it just once. Further, it will stop on the first match, so it will do fewer matching operations when the line matches (in general -- unless the last pattern is the only one to match) and again saves effort by the computer. Modern computers are fast; this is likely a second order effect, especially if you have many patterns and few of them are ever matched.

Jonathan Leffler Over a year ago

Well, neither $0 (what you are storing) nor NR (where you are storing) changes, so you'd copy the same material multiple times. Note that you will, in general, have a sparse array because not every line will match. If you wanted a copy of the line every time you matched, then you'd use something like a_rem[n++] = $0 (probably with the break). If you need to know which pattern matched the line, then you have to save that as well (a_pat[NR] = pat, or a_pat[n] = pat, but be careful where you increment n). It's a programming language; you can do lots of variations very easily.

Jonathan Leffler Over a year ago

As long as you're aware that the elements need not be served in any specific order, that's fine.

|

Collectives™ on Stack Overflow

traverse array sent from bash, in external awk script

1 Answer 1

Shell

Awk

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Shell

Awk

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related