Embedding awk in a shell script

Question

I've been using a bash script (script.sh) which uses various awk scripts (script1.awk,script2.awk) which are tailored at "runtime", by replacing values for instance. I've been looking for ways to embed them completely within the first bash script. Ideally, I would like to have a file looking like this :

################################
# AWK scripts                  #
################################
read -d '' scriptVariable <<'EOF'
'
{my block commands;}
'
EOF
################################
# End of AWK Scripts           #
################################
awk $scriptVariable ${inputfile} # This line obviously doesn't work

instead of the traditional :

awk '{
my script commands
' ${intputfile}

Of course, I could write them to a file but the whole point is not to. Any suggestions ?

EDIT : Although dogbane answers works fine, the next problem is that with the <<'HERE' tags, newline characters are not read.I can't unquote it since, otherwise, he's trying to interpret the awk script then encountering a $ sign within (and there are). And with no newlines, I can't comment anything within the awk script (without commenting half the script when the newlines characters are being removed ...). Anyone ?

<< 'EOF'
BEGIN{#Hello
print $1
}
EOF # Is read as BEGIN{#Hello print $1} by awk; ie BEGIN{

<< EOF
BEGIN{#Hello
print $1
}
EOF #Is read correctly by awk but Bash tried to express $1 and fails

It sounds like you simply need to upgrade your scripting language to something better - Perl or Python. It will be easier and work faster. — mvp
– mvp, Commented Feb 22, 2013 at 8:45
Already tried, it failed with awk: syntax error at source line 1 context is >>> ' <<< awk: bailing out at source line 18 — Bertrand Caron
– Bertrand Caron, Commented Feb 22, 2013 at 8:47
why are you reading the variable when you can just set it with scriptVar='...' or better yet skip the whole setting of a variable and just use awk ' ... ' file ? — technosaurus
– technosaurus, Commented Feb 22, 2013 at 9:05
@technosaurus 1.Because there are nasty characters (',",..) in my string 2.Because I want to segregate the bash code and the awk code. — Bertrand Caron
– Bertrand Caron, Commented Feb 22, 2013 at 9:18

dogbane · Accepted Answer · 2013-02-22 09:12:17Z

19

Remove the single quotes around the awk script and enclose the script variable in double-quotes. This works for me:

################################
# AWK scripts                  #
################################
read -d '' scriptVariable << 'EOF'
BEGIN {
    print "start"
}
{
    print $0
}
END{
    print "hello"
}
EOF
################################
# End of AWK Scripts           #
################################

awk "$scriptVariable" ${inputfile}

answered Feb 22, 2013 at 9:12

dogbane

276k77 gold badges407 silver badges415 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

sdaigle Over a year ago

To pass shell vars to the Awk script, use Awk's "-v" option. Something like: var v1 = "shell value" awk -v awkVar="$v1" "$scriptVariable" ${inputfile} In your script, 'awkVar' will hold the value "shell value"

technosaurus · Accepted Answer · 2013-02-22 09:18:14Z

2

The single quotes around awk scripts can be cantankerous in a shell.

echo dummy code | awk '/dummy/{print '$1' $2}' - another_file | while read LINE; do
    case $LINE in
        "$1 code")echo success;;
        *)echo $LINE;;
    esac
done

This will remove from stdin and then another_file and output lines containing dummy and replace the first field with the first argument to your bash script. Notice that the bash variable $1 had to be unquoted.

answered Feb 22, 2013 at 9:18

technosaurus

7,8621 gold badge33 silver badges53 bronze badges

Comments

Axel · Accepted Answer · 2013-02-22 08:54:19Z

1

You can include your awk inside your bash script using a Here-Document.

answered Feb 22, 2013 at 8:54

Axel

14.2k6 gold badges55 silver badges80 bronze badges

3 Comments

cdarke Over a year ago

A tip on that, use the -f - option, as in awk -f - << EOD

user674669 Over a year ago

Why is the second - required?

user674669 Over a year ago

Bruno's answer at superuser.com/questions/365452/how-to-write-awk-here-document worked for me.

Collectives™ on Stack Overflow

Embedding awk in a shell script

3 Answers 3

1 Comment

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related