jq - create empty array and add objects to it

Question

I am working on a bash script (using jq for JSON parsing)

that needs to make multiple CURL calls (response has same structure but different values), apply some logic/filters and then collate all the responses in one final JSON array of objects.
Loop through this final JSON array and write into a CSV in a predefined format.

Searched for a bit for both the requirements but could not find anything concrete. Please advice. The highlighted steps below (in ***) are the points where I need help.

Sample Flow :

create empty FINAL array

for(eachService is serviceList)
       a. CURL <service_response> returning JSON array of objects
       b. use jq filters to parse JSON response, apply some logic and modify elements in response as needed
       c. ***add this JSON array to FINAL array***

***LOOP through FINAL array, one object at a time a write to CSV.***

Sample Data :

CURL Response 1 (ex: $curl1):
[
  {
    "id":"123",
    "startDate": "2016-12-09T00:00:00Z",
    "calls":4
  },
  {
    "id":"456",
    "startDate": "2016-12-09T00:00:00Z",
    "calls":22
  }
]

CURL Response 2 (ex : $curl2): 
[
  {
    "id":"789",
    "startDate": "2016-12-09T00:00:00Z",
    "calls":8
  },
  {
    "id":"147",
    "startDate": "2016-12-09T00:00:00Z",
    "calls":10
  }
]

NEEDED OUTPUT ($final): 
[
{
    "id":"123",
    "startDate": "2016-12-09T00:00:00Z",
    "calls":4
  },
  {
    "id":"456",
    "startDate": "2016-12-09T00:00:00Z",
    "calls":22
  },
  {
    "id":"789",
    "startDate": "2016-12-09T00:00:00Z",
    "calls":8
  },
  {
    "id":"147",
    "startDate": "2016-12-09T00:00:00Z",
    "calls":10
  }
]

hek2mgl · Accepted Answer · 2016-12-19 07:17:31Z

3

jq can deal with multiple input arrays. You can pipe the whole output of the loop to it:

for service in "$services" ; do
    curl "$service/path"
done | jq -r '.[]|[.id,.startDate,.calls]|@csv'

Note that the csv transformation can be done by @csv

edited Dec 19, 2016 at 7:17

answered Dec 19, 2016 at 7:09

hek2mgl

159k31 gold badges263 silver badges279 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Vinay Over a year ago

Is it possible to also sort the rows from across the calls ? For example, I'd want the CSV rows to be sorted on the descending order of 'calls' value. Thanks

hek2mgl Over a year ago

Pipe it to | sort -t',' k3,3nr

peak · Accepted Answer · 2016-12-19 15:23:17Z

2

As @hek2mlg pointed out, it should be possible to invoke jq just once. If the input is sufficiently uniform (admittedly, maybe a big "if"), you could even avoid having to name the fields explicitly, e.g.:

$ for service in "$services" ; do
    curl "$service/path"
  done | jq -sr 'add[] | [.[]] | @csv'

Output:

"123","2016-12-09T00:00:00Z",4
"456","2016-12-09T00:00:00Z",22
"789","2016-12-09T00:00:00Z",8
"147","2016-12-09T00:00:00Z",10

Note that using -s allows you to perform arbitrary computations on all the inputs, e.g. counting them.

edited Dec 19, 2016 at 15:23

answered Dec 19, 2016 at 8:11

peak

119k21 gold badges185 silver badges218 bronze badges

Comments

janos · Accepted Answer · 2016-12-19 06:54:51Z

1

You don't need a "final" array. You can process the individual JSON output of each curl call, and parse each with the same jq script, converting JSON input to CSV output. Something along the lines of:

for url; do
    curl "$url" | jq 'filter-and-extract-csv-columns'
done > output.csv

Notice the redirection of the entire loop to output.csv.

This kind of streamlined processing is possible thanks to the CVS format being flat without a surrounding context as in XML or JSON. The output of multiple computations can be simply concatenated.

answered Dec 19, 2016 at 6:54

janos

126k31 gold badges242 silver badges253 bronze badges

4 Comments

Vinay Over a year ago

also need to address the requirement of data transformation (add/modify) before writing to CSV. Was hence trying to do it JSON and jq.

janos Over a year ago

Well yes, do all the necessary data transformations in jq, and then output in CSV. You don't need an intermediary JSON.

Vinay Over a year ago

qq - If we write to the same CSV file for all the calls, does it add the headers multiple times ?

janos Over a year ago

If you need CSV headers, you can write the header line before the loop.

Collectives™ on Stack Overflow

jq - create empty array and add objects to it

3 Answers 3

2 Comments

Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related