Parsing JSON from shell script using JSON.sh

I wrote a JSON serializer / deserializer for gawk, if you're interested. Save that script and modify it, replacing everything above # === FUNCTIONS === with the following:

#!/usr/bin/gawk -f

# capture JSON string from beginning to end into a scalar variable
{ json = json ORS $0 }

END {
    # objectify JSON string to the multilevel array "obj"
    deserialize(json, obj)

    for (filename in obj) {

        print "file_name=" quote(filename)

        for (key in obj[filename]) {

            # print key="value"
            print key "=" quote(obj[filename][key])
        }
    }
}

Do chmod 755 json.awk and execute it. Output will resemble this:

$ ./json.awk test5.json
file_name="/home/ukrishnan/projects/mysql/app.xml"
ENV_ACCOUNT_BRIDGE_ENDPOINT="/u01/src/test/sample.txt"
file_name="/home/ukrishnan/projects/test.yml"
LOG_DRIVER="syslog"
IMAGE="mysql:5.6"

Hopefully the logic is reasonably easy to follow. If you prefer to output filename=, key=, and value= on every loop iteration, modify the nested for loops accordingly:

for (filename in obj) {
    for (key in obj[filename]) {
        print "file_name=" quote(filename)
        print "key=" quote(key)
        print "value=" quote(obj[filename][key])
    }
}

That change will result in the following output:

$ ./json.awk test5.json
file_name="/home/ukrishnan/projects/mysql/app.xml"
key="ENV_ACCOUNT_BRIDGE_ENDPOINT"
value="/u01/src/test/sample.txt"
file_name="/home/ukrishnan/projects/test.yml"
key="LOG_DRIVER"
value="syslog"
file_name="/home/ukrishnan/projects/test.yml"
key="IMAGE"
value="mysql:5.6"

Anyway, with that output, you can do something silly in BASH like this to populate and act upon the variables:

#!/bin/bash

./test.awk test5.json | while read -r line; do {
    eval $line
    [ "${line/=*/}" = "value" ] && {
        echo "bash: file_name=$file_name"
        echo "bash: key=$key"
        echo "bash: value=$value"
        echo "------"
    }
}; done

It'd probably be more graceful just to do all processing within gawk from start to finish and not mess with the polyglot handoff, though.

Getting back to json.awk, if you prefer to keep json.awk modular for easy reuse in future projects, you could remove everything above # === FUNCTIONS ===, create a separate main.awk containing the code block at the top of this answer, and @include "json.awk" as a helper library pretty much anywhere outside of END {...} (just below the shbang, for example).

JSON.sh (from http://json.org) offers a nice bash friendly means of flattening out a JSON file. Which you've already provided how it looks in your question. So, the flatten form is the format:

[node] tab value

You have to think in UNIX script in extracting the information you want, you'll note the lines you're interested in actually follow this pattern:

• ["filename","key"] tab ["value"]

In regex notation, we replace:

• filename with (.*)
• key with (.*)
• tab with \t
• value with (.*)

We can retrieve the first, second and third matching groups with \1, \2, \3 respectively.

When used in sed we also note that these symbols []() need to be escaped with a backslash \, resulting in the following script:

./lib/JSON.sh < test/test.json | sed 's/\["\(.*\)","\(.*\)\"]\t"\(.*\)"/\1,\2,\3/;t;d'
/home/ukrishnan/projects/test.yml,LOG_DRIVER,syslog
/home/ukrishnan/projects/test.yml,IMAGE,mysql:5.6
/home/ukrishnan/projects/mysql/app.xml,ENV_ACCOUNT_BRIDGE_ENDPOINT,/u01/src/test/sample.txt

Now we put the lines in a loop and for each line, we can extract out filename,key,value:

for line in $(./lib/JSON.sh < test/test.json | sed 's/\["\(.*\)","\(.*\)\"]\t"\(.*\)"/\1,\2,\3/;t;d')
do
  IFS="," read -ra arr <<< $line
  filename=${arr[0]}
  key=${arr[1]}
  value=${arr[2]}
  cat <<EOF
filename : $filename
key      : $key
value    : $value
EOF
done

Which outputs:

filename : /home/ukrishnan/projects/test.yml
key      : LOG_DRIVER
value    : syslog
filename : /home/ukrishnan/projects/test.yml
key      : IMAGE
value    : mysql:5.6
filename : /home/ukrishnan/projects/mysql/app.xml
key      : ENV_ACCOUNT_BRIDGE_ENDPOINT
value    : /u01/src/test/sample.txt