Parse JSON data in variables using JQ

Question

I have some problems with my script which collects data from a JSON, stores them in variables which are then used for a CURL request. I need to build a CURL request for EACH JSON entry.

My problem is that I would like to pass parameters to the CURL request, one by one. I was thinking about a for-loop but this won't be actually the right workaround.

This because

ruleId=$($whitelist | jq -r '.[].ruleId')

gives:

10055
10098

This can not be interpreted correctly from CURL.

So the question is, how can I pass variables in a proper manner, in a sort of iteration, using JQ? Again, I need to do single calls using CURLs for each entry in the JSON file.

Code:

$!/bin/sh
#set -e

# Info to test this script
# 1. Start docker 
# 2. Run this command from terminal: docker run -u zap -p 8080:8080 -i owasp/zap2docker-stable zap.sh -daemon -host 0.0.0.0 -port 8080 -config api.disablekey=true -config api.addrs.addr.name=.* -config api.addrs.addr.regex=true
# 2. Get the JSON from api (test for now)
# 3. bash filter.sh
# 4. Check for filter to be set with browser http://localhost:8080/UI/alertFilter/ => view global filter

curl -s 'https://api.npoint.io/c29e3a68be632f73fc22' > whitelist_tmp.json
whitelist="cat whitelist_tmp.json"
listlength=$(jq '. | length' $whitelist)


ruleId=$($whitelist | jq -r '.[].ruleId')
alert=$($whitelist | jq -r '.[].alertName')
newLevel=$($whitelist | jq -r '.[].newLevel')
url=$($whitelist | jq -r '.[].url | .[]')
urlIsRegex=$($whitelist | jq -r '.[].urlIsRegex')
enabled=$($whitelist | jq -r '.[].enabled')
parameter=$($whitelist | jq -r '.[].parameter')
evidence=$($whitelist | jq -r '.[].evidence')
echo "Setting Rule for: $ruleId"
echo "$(curl --data-urlencode "ruleId=$ruleId" --data-urlencode "newLevel=$newLevel" --data-urlencode "url=$url" --data-urlencode "urlIsRegex=$urlIsRegex" --data-urlencode "enabled=$enabled" --data-urlencode "parameter=$parameter" --data-urlencode "evidence=$evidence" "http://localhost:8090/JSON/alertFilter/action/addGlobalAlertFilter")"

Renaud Pacalet · Accepted Answer · 2021-08-10 14:13:27Z

2

You can use bash arrays to store your values:

ruleId=($($whitelist | jq -r '.[].ruleId'))
alert=($($whitelist | jq -r '.[].alertName'))
...

and then iterate over them. Example:

for (( i = 0; i < "${#ruleId[@]}"; i++ )); do
  id="${ruleId[i]}"
  al="${alert[i]}"
  ...
  echo "$(curl --data-urlencode "ruleId=$id" ...
done

This works if and only if the values returned by your commands are single words (no spaces in them) or they are properly quoted. If you have more complex values you cannot simply assign them to an array with array=($(command)). You would get more cells than values in your array.

edited Aug 10, 2021 at 14:13

answered Aug 10, 2021 at 14:07

Renaud Pacalet

30.7k3 gold badges42 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

hobbs Over a year ago

jq has an @sh formatter. It will do any necessary quoting for you and produce output suitable for assignment to a shell array.

t30_9 Over a year ago

Is it actually possibile that this solution could affect a next script? For explanation: i'm trying now to run this in Gitlab, pipeline. Running the script using this structure, causes problems to the next script. This, even if there are no dependecies.

Renaud Pacalet Over a year ago

@mdem99 Without knowing what your scripts do and how they interact in your CI setup it is impossible to answer this question, sorry. Open a new question, maybe, with all the needed information about the scripts and your .gitlab-ci.yml.

markp-fuso · Accepted Answer · 2021-08-10 15:07:13Z

1

Pulling the comments from your previous question:

whitelist="whitelist_tmp.json"
listlength=$(jq '. | length' "${whitelist}")

mapfile -t rule      < <(jq -r '.[].ruleId'      "${whitelist}")
mapfile -t alert     < <(jq -r '.[].cwalertName' "${whitelist}")
mapfile -t level     < <(jq -r '.[].newLevel'    "${whitelist}")
mapfile -t url       < <(jq -r '.[].url | .[]'   "${whitelist}")
mapfile -t regex     < <(jq -r '.[].urlIsRegex'  "${whitelist}")
mapfile -t parameter < <(jq -r '.[].parameter'   "${whitelist}")
mapfile -t evidence  < <(jq -r '.[].evidence'    "${whitelist}")

for ((i=0; i<${listlength}; i++))
do
    curl ... "${rule[$i]}" ... "${alert[$i]}" ...
done

The mapfile should maintain embedded white space in values returned by jq.

edited Aug 10, 2021 at 15:07

answered Aug 10, 2021 at 15:02

markp-fuso

38.5k5 gold badges24 silver badges48 bronze badges

3 Comments

t30_9 Over a year ago

Interesting solution. Any particular advantage of using mapfile instead of JQ directly?

markp-fuso Over a year ago

I'm a novice with jq so can't really answer that question; if jq has the option to designate which entry to extract (eg, 1st, 2nd, ... nth) then sure that would work, too ... then again, you're not using such an option in your code so I assumed such an option doesn't exist ... ???

t30_9 Over a year ago

Actually yes, there is the possibilty to do that but it depends strongly on how your JSON is designed. In my case, the previous solution is really good :)

Collectives™ on Stack Overflow

Parse JSON data in variables using JQ

2 Answers 2

3 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related