awk array pattern not matching

Question

Here is my Shell script

#!/bin/bash
# need this or can't process file names with spaces
OIFS="$IFS"
IFS=$'\n'
file_name=file_name.xlsx

# --re-interval or the repeat character {} will not work
python ./Enforcer/scripts/xlxstocsv.py -m -s 1 -d ';' $file_name | awk --re-interval -F ';' '
# list of all the CGIDs from the schema
BEGIN {split("EKM-04.3 EKM-03.1 AAC-02.5 DCS-02.1 BCR-11.1 IAM-13.2 HRS-01.1 IVS-03.1 TVM-02.5 IAM-10.1 DSI-01.4 DSI-03.1 DSI-05.1 AAC-02.6 IAM-04.1 BCR-11.2 IAM-13.3 GRM-06.2 IAM-01.1 AIS-01.2 DSI-02.2 IVS-07.1 SEF-02.1 BCR-05.1 EKM-03.3 AAC-02.7 AAC-03.2 IAM-11.1 HRS-03.1 GRM-01.1 AAC-03.1 HRS-08.2 IAM-08.1 AAC-02.3 DCS-07.1 AIS-02.1 AAC-03.3 IAM-11.2 BCR-11.4 SEF-04.3 EKM-02.1 SEF-03.1 DSI-07.1 BCR-10.1 BCR-06.1 AIS-03.1 HRS-04.1 DSI-01.7 IVS-01.2 IAM-06.1 BCR-11.5 SEF-04.4 SEF-03.2 DSI-07.2 AIS-01.5", schema_CGID) }
{
# 
if (match($3,/[A-Z]{2,3}-[0-9]{2}\.[0-9]/))
    {
# use the index to find the location of the patten
    let_it_be=index($3,substr($3,RSTART,RLENGTH))
# the CGID seems to USUALLY be in the first 30 spaces of the output
    if (let_it_be < 30) 
        {
        CGID=substr($3,RSTART,RLENGTH)
        { for (N=1; N<=NF; N++) {if (schema_CGID[$N] ~ CGID) {print "schema match ", $i} else  {print "no schema match for CGID [" CGID "] schema [" schema_CGID[$N] "]", $N}}}
        if (match(tolower($0),/yes|no/))
            {
            yes_no=substr($0,RSTART,RLENGTH)
            }
        else {yes_no="NA"}
            }
    foobar[CGID " " yes_no]
    }
}
# the sort is necessary for the output to be useful, as the array uses weird things and the order is lost
END{ for (var in foobar) print var | "sort"}'
#END{ for (var in foobar) print var }'
IFS="$OIFS"

I am processing a XLSX (Excel spreadsheet) file with a Python tool xlxstocsv.py. I've (tried to) create an array with this string

BEGIN {split("EKM-04.3 EKM-03.1 AAC-02.5 DCS-02.1 BCR-11.1 IAM-13.2 HRS-01.1 IVS-03.1 TVM-02.5 IAM-10.1 DSI-01.4 DSI-03.1 DSI-05.1 AAC-02.6 IAM-04.1 BCR-11.2 IAM-13.3 GRM-06.2 IAM-01.1 AIS-01.2 DSI-02.2 IVS-07.1 SEF-02.1 BCR-05.1 EKM-03.3 AAC-02.7 AAC-03.2 IAM-11.1 HRS-03.1 GRM-01.1 AAC-03.1 HRS-08.2 IAM-08.1 AAC-02.3 DCS-07.1 AIS-02.1 AAC-03.3 IAM-11.2 BCR-11.4 SEF-04.3 EKM-02.1 SEF-03.1 DSI-07.1 BCR-10.1 BCR-06.1 AIS-03.1 HRS-04.1 DSI-01.7 IVS-01.2 IAM-06.1 BCR-11.5 SEF-04.4 SEF-03.2 DSI-07.2 AIS-01.5", schema_CGID)

I'm processing my file OK, but, I want to test if the string I am successfully finding is in this schema_CGID array The variable that I'm successfully creating is OK,and I've tried many things, but this is the closest I get

I can even see a match (3rd line)!

no schema match for CGID [AIS-01.1] schema [] Application Security"
no schema match for CGID [AIS-01.1] schema [] AIS-01
no schema match for CGID [AIS-01.1] schema [] AIS-01.1
no schema match for CGID [AIS-01.1] schema [] Applications and programming interfaces (APIs) shall be designed, developed, deployed, and tested in accordance with leading industry standards (e.g., OWASP for web applications) and adhere to applicable legal, statutory, or regulatory compliance obligations.
no schema match for CGID [AIS-01.1] schema [] Do you use industry standards (Build Security in Maturity Model [BSIMM] benchmarks, Open Group ACS Trusted Technology Provider Framework, NIST, etc.) to build in security for your Systems/Software Development Lifecycle (SDLC)?
no schema match for CGID [AIS-01.1] schema [] Yes
no schema match for CGID [AIS-01.1] schema [] 
no schema match for CGID [AIS-01.1] schema []

why is my if not matching my array item? The value of schema_CGID[$N] appears to be valid, but the comparison seems to fail

TIA

Looks like your schema_CGID[$N] value has a leading space in in from that output I think. Try with print "no schema match for CGID [" CGID "] schema [" schema_CGID[$N] "]" instead to see? (Also providing sample input helps tremendously with getting useful help.) — Etan Reisner
– Etan Reisner, Commented May 13, 2016 at 2:38
I dont think this is the right syntax for an if else block in bash..try this from the terminal . example : echo "foo bar bam" | if [[ $(awk '{printf $3}') == "bam" ]]; then echo "found bam" ; else echo "bam does not existecho" ; fi found bam — z atef
– z atef, Commented May 13, 2016 at 2:45

Ed Morton · Accepted Answer · 2016-05-13 04:47:04Z

2

split() uses the FS value by default. You set FS to ; on the command line but then the string you're trying to split is separated by spaces so the resultant array schema_CGID only contains 1 value - the whole string you're trying to split. Make it split("...",schema_CGID,/ /).

answered May 13, 2016 at 4:47

Ed Morton

209k18 gold badges90 silver badges212 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Henry Barber Over a year ago

thanks @Ed - now my for loop is failing because of the same short-sightedness my comparison of the schema_CGID array item to the CGID fails because the N counter stops (as it is written to dos so!) too early - again, because of the FS, so doesn't reach the value and fails

Henry Barber Over a year ago

what I'm doing is (incorrectly) looping through each field of the input; what I want to do is loop through each line of my schema_CGID array to see if CGID is present.

Henry Barber Over a year ago

if have tried { if (CGID in schema_CGID) ; print "in ",CGID } but it outputs every CGID as being present. Any clues how I test the presence of that variable in my array ?

Ed Morton Over a year ago

get rid of the semi-colon thats terminating the if and making the print unconditional: { if (CGID in schema_CGID) print "in ",CGID }

Collectives™ on Stack Overflow

awk array pattern not matching

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related