how to for loop a jq array result in shell script

Question

I have a json data as below

{
  "id": {
    "bioguide": "E000295",
    "thomas": "02283",
    "govtrack": 412667,
    "opensecrets": "N00035483",
    "lis": "S376"
  },
  "bio": {
    "gender": "F",
    "birthday": "1970-07-01"
  },
  "tooldatareports": [
    {
      "name": "A",
      "tooldata": [
        {
          "toolid": 12345,
          "data": [
            {
              "time": "2021-01-01",
              "value": 1
            },
            {
              "time": "2021-01-02",
              "value": 10
            },
            {
              "time": "2021-01-03",
              "value": 5
            }
          ]
        },
        {
          "toolid": 12346,
          "data": [
            {
              "time": "2021-01-01",
              "value": 10
            },
            {
              "time": "2021-01-02",
              "value": 100
            },
            {
              "time": "2021-01-03",
              "value": 50
            }
          ]
        }
      ]
    }
  ]
}

now I can use below command line to get a list containing two dict, each dict have a key "data", value is a list,

cat data.json |jq -n --stream '[fromstream(inputs | (.[0] | index("data")) as $ix | select($ix) | .[0] |= .[$ix:])]'

I want to print each dict with a loop in shell script

My except in each loop have printed a dict, total have 2 dict

But it seems like a string

below is my shell script

array=$(cat ernst.json | jq -n --stream '[fromstream(inputs | (.[0] | index("data")) as $ix | select($ix) | .[0] |= .[$ix:])]')

for d in $array
do
    echo $d
done

Does anyone have any idea?

William Pursell · Accepted Answer · 2021-09-04 09:46:55Z

3

$ jq -cn --stream '[fromstream(inputs | (.[0] | index("data")) as $ix | select($ix) | .[0] |= .[$ix:])][]' data.json | while read d; do echo "item: $d"; done
item: {"data":[{"time":"2021-01-01","value":1},{"time":"2021-01-02","value":10},{"time":"2021-01-03","value":5}]}
item: {"data":[{"time":"2021-01-01","value":10},{"time":"2021-01-02","value":100},{"time":"2021-01-03","value":50}]}

Note that you can get very similar output with simpler jq:

jq -c '.tooldatareports[].tooldata[].data' data.json  | while read d; do echo "item: $d"; done
item: [{"time":"2021-01-01","value":1},{"time":"2021-01-02","value":10},{"time":"2021-01-03","value":5}]
item: [{"time":"2021-01-01","value":10},{"time":"2021-01-02","value":100},{"time":"2021-01-03","value":50}]

And you can get identical output (this seems likely to be unnecessary) with:

$ jq -c '.tooldatareports[].tooldata[].data | {"data": .}' data.json  | while read d; do echo "item: $d"; done
item: {"data":[{"time":"2021-01-01","value":1},{"time":"2021-01-02","value":10},{"time":"2021-01-03","value":5}]}
item: {"data":[{"time":"2021-01-01","value":10},{"time":"2021-01-02","value":100},{"time":"2021-01-03","value":50}]}

edited Sep 4, 2021 at 9:46

answered Sep 4, 2021 at 9:35

William Pursell

214k49 gold badges279 silver badges317 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

CYC Over a year ago

nice method, but I need to use --stream flag because my final target file is huge

swingbit · Accepted Answer · 2021-09-04 09:39:31Z

2

Your output is a json list, so you won't get away with a direct bash loop over it. The bash loop won't know what to do with the square brackets and will use each space and newline as a separator. Use jq again to get separate objects from your json list. That is done with jq -c '.[]':

array=$(cat data.json |jq -n --stream '[fromstream(inputs | (.[0] | index("data")) as $ix | select($ix) | .[0] |= .[$ix:])]' | jq -c '.[]')

Now you have two objects in two separate lines and no spaces (-c = compact output) and you can loop over them in bash:

for d in $array
do
    echo "$d"
    # whatever else you need to do with them
done

{"data":[{"time":"2021-01-01","value":1},{"time":"2021-01-02","value":10},{"time":"2021-01-03","value":5}]}
{"data":[{"time":"2021-01-01","value":10},{"time":"2021-01-02","value":100},{"time":"2021-01-03","value":50}]}

edited Sep 4, 2021 at 9:39

answered Sep 4, 2021 at 9:27

swingbit

2,76520 silver badges38 bronze badges

3 Comments

Philippe Over a year ago

You did two calls to jq. Can it be done with a single call ?

swingbit Over a year ago

@Philippe yes, you can just add -c and [] to the first jq invocation. I don't always prefer the most compact solution though. I often prefer a clear separation of concerns instead. That is what pipelines are great for. In this case, I thought it was more important to show the OP what each step does.

CYC Over a year ago

very clear to explain the bash loop, thanks

Collectives™ on Stack Overflow

how to for loop a jq array result in shell script

2 Answers 2

1 Comment

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related