adding item to json using bash variable with jq

Question

I am trying to use bash to create a string variable that will be used in jq to add new elements to the json file. But it's escaping my double quote. See below for "wrong output" produced. I am expecting the output shown below "expected output". What is the correct way to add more fields to filename with bash variable?

My input json file (input.json):

{
  "##_Comment1": "Inputs",
  "filename": [
    "file1",
    "file2",
    "file3",
    "file4"
  ]
}

My bash script:

#!/bin/bash
update_list='"abc","efg"'

cat input.json | jq --arg args "$update_list" '.["filename"] += [$args]'

wrong output:

{
  "##_Comment1": "Inputs",
  "filename": [
    "file1",
    "file2",
    "file3",
    "file4",
    "\"abc\",\"efg\""
  ]
}

correct output:

{
  "##_Comment1": "Inputs",
  "filename": [
    "file1",
    "file2",
    "file3",
    "file4",
    "abc",
    "efg"
  ]
}

Not familiar with jq, just a shot in the dark. The problem seems to be that it reads the update_list as string and not as array so maybe update_list=("abc","efg") and pass update_list to jq without double quotes — Theofanis
– Theofanis, Commented Jul 30, 2018 at 13:51
@Theofanis, that's not a 2-item bash array, because , does not separate array items in bash -- update_list=( abc efg ) would be a native bash array. That said, it would be saner to pass the list as a JSON string itself: update_list='["abc","efg"]' — Charles Duffy
– Charles Duffy, Commented Jul 30, 2018 at 17:09

Attie · Accepted Answer · 2018-07-30 14:05:36Z

2

Unwind your situation a bit:

$ jq --arg args '"abc","efg"' '.["filename"] += [$args]' <<< '{"filename":[]}'
{
  "filename": [
    "\"abc\",\"efg\""
  ]
}

Here, we are effectively assigning args to a string in the jq engine:

args = "\"abc\",\"efg\""

If you want set args to a list/array, then you'll need to take another approach.

You could either format a JSON argument and use --argjson:

$ jq --argjson args '["abc","efg"]' '.["filename"] += $args' <<< '{"filename":[]}'
{
  "filename": [
    "abc",
    "efg"
  ]
}

Or you could make update_list into an array, and loop:

update_list=()
update_list+=("abc")
update_list+=("efg")

echo '{"filename":[]}' > test

for i in "${update_list[@]}"; do
    jq --arg update_item "${i}" '.["filename"] += [ $update_item ]' < test \
        | sponge test
done

$ cat test
{
  "filename": [
    "abc",
    "efg"
  ]
}

edited Jul 30, 2018 at 14:05

answered Jul 30, 2018 at 13:58

Attie

7,0972 gold badges31 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Charles Duffy Over a year ago

For the last case, I would consider --slurpfile to read the input file as a variable, and printf '%s\n' "${update_list[@]}" to serialize the list into a line-oriented string (if we don't need to worry about list elements containing literal newlines) which can be fed on stdin; that way, jq -Rn will be able to get that list as inputs.

peak · Accepted Answer · 2018-07-31 00:37:59Z

1

In the original question, update_list is a comma-separated listing of quoted strings, and the following solution would work if these strings are also JSON strings:

jq --arg args "$update_list" '
  .["filename"] += ($args|split(",")|map(fromjson))'

update_list variants

If update_list can be made available as a JSON array, then @Attie's first solution would be the way to go.

However if update_list is a bash array, the solution involving one call to jq per array element is unnecessarily (and might be embarrassingly) inefficient; the suggested solution might also create problems because the update is not atomic. There are far better alternatives. For example, the jq FAQ mentions a technique which, when applied to the present problem, yields the following solution:

jq --argjson args "$(printf '%s\n' "${update_list[@]}" | jq -nR '[inputs]')" '
 .["filename"] += $args'

Or for robustness, one could use NUL as the delimiter:

jq --argjson args "$(printf '%s\0' "${update_list[@]}" | jq -sR 'split("\u0000")')" '
 .["filename"] += $args'

See also jq convert bash array to json array and insert to file

edited Jul 31, 2018 at 0:37

answered Jul 30, 2018 at 20:35

peak

119k21 gold badges185 silver badges218 bronze badges

1 Comment

peak Over a year ago

@Charles - Revised.

Collectives™ on Stack Overflow

adding item to json using bash variable with jq

2 Answers 2

1 Comment

update_list variants

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

update_list variants

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related