Add new element to existing JSON array with jq

Question

I want to append an element to an array in a JSON file using the jq``add command, but it's not working.

report-2017-01-07.json file:

{  
   "report": "1.0",
   "data": {  
      "date": "2010-01-07",
      "messages": [  
         {  
            "date": "2010-01-07T19:58:42.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "OK",
            "message": "metadata loaded into iRODS successfully"
         },
         {  
            "date": "2010-01-07T20:22:46.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "NOK",
            "message": "metadata duplicated into iRODS"
         },
         {  
            "date": "2010-01-07T22:11:55.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "NOK",
            "message": "metadata was not validated by XSD schema"
         }
      ]
   }
}

I am using this command:

$ cat report-2017-01-07.json 
| jq -s '.data.messages {"date": "2010-01-07T19:55:99.999Z", "xml": "xml_samplesheet_2017_01_07_run_09.xml", "status": "OKKK", "message": "metadata loaded into iRODS successfullyyyyy"}'
jq: error: syntax error, unexpected '{', expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.data.messages {"date": "2010-01-07T19:55:99.999Z", "xml": "xml_samplesheet_2017_01_07_run_09.xml", "status": "OKKK", "message": "metadata loaded into iRODS successfullyyyyy"}               
jq: 1 compile error

Here's how I want the output to look:

{
    "report": "1.0",
    "data": {
        "date": "2010-01-07",
        "messages": [{
            "date": "2010-01-07T19:58:42.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "OK",
            "message": "metadata loaded into iRODS successfully"
        }, {
            "date": "2010-01-07T20:22:46.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "NOK",
            "message": "metadata duplicated into iRODS"
        }, {
            "date": "2010-01-07T22:11:55.949Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "NOK",
            "message": "metadata was not validated by XSD schema"
        }, {
            "date": "2010-01-07T19:55:99.999Z",
            "xml": "xml_samplesheet_2017_01_07_run_09.xml",
            "status": "OKKKKKKK",
            "message": "metadata loaded into iRODS successfullyyyyy"
        }]
    }
}

Don't know this well but is it not an issue that .data.messages is an arrray? So you need something like .data.messages.[] — Vanquished Wombat
– Vanquished Wombat, Commented Feb 15, 2017 at 9:42
Ok - read the docs and installed jq (on have Windows here) and tried - get same error as you. Looks like Windows shell has issues with the way it feeds double quotes into the stream which trips up jq. Could not get anything to work so no answer to your question but you might want to look at escaping the double quotes in the jq command. So "date" would become \"date\" etc. — Vanquished Wombat
– Vanquished Wombat, Commented Feb 15, 2017 at 10:11
if I use this command I can update all the date attributes. jq '.data.messages[].date = "2010-01-07T99:99:99.999Z"' report-2017-01-07.json but I cannot add yet. — Felipe
– Felipe, Commented Feb 15, 2017 at 10:41

Schlueter · Accepted Answer · 2022-12-13 13:46:07Z

190

The |= .+ part in the filter adds a new element to the existing array. You can use jq with filter like:

jq '.data.messages[3] |= . + {
      "date": "2010-01-07T19:55:99.999Z", 
      "xml": "xml_samplesheet_2017_01_07_run_09.xml", 
      "status": "OKKK", 
      "message": "metadata loaded into iRODS successfullyyyyy"
}' inputJson

To avoid using the hardcoded length value 3 and dynamically add a new element, use . | length which returns the length, which can be used as the next array index, i.e.,

jq '.data.messages[.data.messages| length] |= . + {
      "date": "2010-01-07T19:55:99.999Z", 
      "xml": "xml_samplesheet_2017_01_07_run_09.xml", 
      "status": "OKKK", 
      "message": "metadata loaded into iRODS successfullyyyyy"
}' inputJson

(or) as per peak's suggestion in the comments, using the += operator alone

jq '.data.messages += [{
     "date": "2010-01-07T19:55:99.999Z",
     "xml": "xml_samplesheet_2017_01_07_run_09.xml", 
     "status": "OKKK", 
     "message": "metadata loaded into iRODS successfullyyyyy"
}]'

which produces the output you need:

{
  "report": "1.0",
  "data": {
    "date": "2010-01-07",
    "messages": [
      {
        "date": "2010-01-07T19:58:42.949Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "OK",
        "message": "metadata loaded into iRODS successfully"
      },
      {
        "date": "2010-01-07T20:22:46.949Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "NOK",
        "message": "metadata duplicated into iRODS"
      },
      {
        "date": "2010-01-07T22:11:55.949Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "NOK",
        "message": "metadata was not validated by XSD schema"
      },
      {
        "date": "2010-01-07T19:55:99.999Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "OKKK",
        "message": "metadata loaded into iRODS successfullyyyyy"
      }
    ]
  }
}

Use jq-play to dry-run your jq-filter and optimize any way you want.

edited Dec 13, 2022 at 13:46

Schlueter

4,1051 gold badge34 silver badges51 bronze badges

answered Feb 15, 2017 at 12:01

Inian

87k15 gold badges166 silver badges183 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

peak Over a year ago

Since the goal is just to append an element, it would be better to use += but it is worth noting that .data.messages[3] |= . + X can here be simplified to .data.messages[3] = X as the . on the RHS is effectively just null.

script'n'code Over a year ago

(windows JQ user here): I wonder if it is possible to write this new data to the same (input) file, instead of creating a temporary file which needs to be renamed afterwords..

peak Over a year ago

@script'n'code - Using sponge is probably still the best option if you have it or can install it (it's part of moreutils).

Nic Huang Over a year ago

@peak how can I insert that object to the beginning of the array, couldn’t figure out . Thanks

Harry Ma Over a year ago

@peak how can I specify the order when inserting the element to the array ?

Rob Bednark · Accepted Answer · 2019-07-31 08:50:42Z

78

Rather than using |=, consider using +=:

.data.messages += [{"date": "2010-01-07T19:55:99.999Z",
   "xml": "xml_samplesheet_2017_01_07_run_09.xml",
   "status": "OKKK", "message": "metadata loaded into iRODS successfullyyyyy"}]

Prepend

On the other hand, if (as @NicHuang asked) you want to add the JSON object to the beginning of the array, you could use the pattern:

 .data.messages |= [ _ ] + .

edited Jul 31, 2019 at 8:50

Rob Bednark

28.8k28 gold badges90 silver badges131 bronze badges

answered Feb 15, 2017 at 18:50

peak

119k21 gold badges185 silver badges218 bronze badges

1 Comment

Vitalii Zinchenko Over a year ago

Prepend example: echo '{"arr": [ "data1" ]}' | jq '.arr |= ["data2"] + .' => {"arr": ["data2", "data1"]}

pr-pal · Accepted Answer · 2019-08-17 14:30:57Z

58

Summary: ". +" is your saviour

Details:

For adding an entry to a list: You can append [list1] + [list2] (and not [list] + data)

$ echo '[ "data1" ]' | jq '. + [ "data2" ]'
[
  "data1",
  "data2"
]

$ echo '[ {"key1": "value1"} ]' | jq '. + [{"key2": "value2"}]'
[
  {
    "key1": "value1"
  },
  {
    "key2": "value2"
  }
]

For adding a key/value to a dictionary:

$ echo '{"key1": "value1"}' | jq '. + {"key2": "value2"}'
{
  "key1": "value1",
  "key2": "value2"
}

References:

https://gist.github.com/joar/776b7d176196592ed5d8

answered Aug 17, 2019 at 14:30

pr-pal

3,62829 silver badges19 bronze badges

Comments

SOFe · Accepted Answer · 2024-09-11 03:13:27Z

An alternative syntax is to iterate the existing values together with the new value(s). This is arguably better or worse:

.data.messages |= [
    values, to, prepend,
    .[],
    values, to append,
]

Alternatively, if .data.messages may be null (nonexistent):

.data.messages |= [
    values, to, prepend,
    select(.)[],
    values, to append,
]

Explanation:

Every "normal" value in jq is actually an iterator that yields 1 item
expr[] creates an iterator that yields all items in expr
value, expr[] joins the two iterators together, i.e. chaining value before expr
select(.) creates an iterator that yields 0 item if . is converted to false, which is the case for null and []. It only yields something when the list is non-empty.
select(.)[] together yields the items if . is a non-null iterable value.

Collectives™ on Stack Overflow

Add new element to existing JSON array with jq

4 Answers 4

5 Comments

Prepend

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

5 Comments

Prepend

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related