Modify a JSON array with inputs from a shell array in only a single pass

Question

Is it possible to filter an entire array of items in JQ in only one pass? Compare the following code, which runs jq over and over:

{
      "foofoo": {
        "barbar": [
          {
            "foo": "aaaa",
            "bar": 0000
          },
          {
            "foo": "bbbb",
            "bar": 1111
          },
          {
            "foo": "cccc",
            "bar": 2222
          }
          ]
      }
}

bash array:

array=("1111" "2222")

my code is working but not very efficient and uses a lot of resources considering the array size in reality:

for k in "${array[@]}"; do 
    jq  --argjson k "$k"  '.foofoo.barbar |= map(select(.bar != $k))' json.json | sponge json.json 
done

It keeps looping through the array, removing the unneeded entries and storing same file again by using sponge.

any ideas how to achieve a similar behavior with a lighter code?

Desired output:

{
  "foofoo": {
    "barbar": [
      {
        "foo": "aaaa",
        "bar": 0
      }
    ]
  }
}

still, you won't get 0000 as output, it will be collapsed to "bar": 0 — RomanPerekhrest
– RomanPerekhrest, Commented Dec 22, 2018 at 17:27
yes it is an integer so 0 or 0000 will not affect the functionality — ma7555
– ma7555, Commented Dec 22, 2018 at 17:46

peak · Accepted Answer · 2018-12-22 18:06:57Z

2

To improve the performance significantly use the following jq approach (without any shell loops):

arr=("1111" "2222")
jq '($p | split(" ") | map(tonumber)) as $exclude 
    | .foofoo.barbar 
      |= map(select(.bar as $b 
                    | any($exclude[]; . == $b) | not))' \
    --arg p "${arr[*]}" file.json | sponge file.json

The output:

{
  "foofoo": {
    "barbar": [
      {
        "foo": "aaaa",
        "bar": 0
      }
    ]
  }
}

edited Dec 22, 2018 at 18:06

peak

119k21 gold badges185 silver badges218 bronze badges

answered Dec 22, 2018 at 16:38

RomanPerekhrest

93.1k4 gold badges75 silver badges112 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

ma7555 Over a year ago

Please check the last edit as .bar value is unquoted in the JSON file

Charles Duffy Over a year ago

@ma7555 That's not valid JSON; strings must be in quotes. Try it yourself: jq . <<<'{"bar": aaaa}' fails with an "invalid numeric literal" error. So does python -m json.tool <<<'{"bar": aaaa}', to show you that that's not just a jq limitation.

ma7555 Over a year ago

@CharlesDuffy it is actually an integer, I resembled it as a string just as an example. I might have caused a confusion with this

Charles Duffy Over a year ago

minimal reproducible example guidelines require that an example be "verifiable" -- it must not cause errors unrelated to the actual question.

RomanPerekhrest Over a year ago

@ma7555, you need to post the actual input avoiding invalid samples

|

glenn jackman · Accepted Answer · 2018-12-22 15:38:03Z

1

I'm positive there are better ways to do this: I really just throw stuff at jq until something sticks to the wall ...

# 1. in the shell, construct a JSON object string from the array => {"bbbb":1,"cccc":1}
printf -v jsonobj '{%s}' "$(printf '"%q":1\n' "${array[@]}" | paste -sd,)"

# 2. use that to test for non-membership in the jq select function
jq --argjson o "$jsonobj" '.foofoo.barbar |= map(select((.bar|in($o)) == false))' json.json

outputs

{
  "foofoo": {
    "barbar": [
      {
        "foo": "0000",
        "bar": "aaaa"
      }
    ]
  }
}

You don't actually show your desired output, so I assume this is what you want.

answered Dec 22, 2018 at 15:38

glenn jackman

249k42 gold badges233 silver badges362 bronze badges

1 Comment

PesaThe Over a year ago

One might also use map(select($o[.bar]!=1)) :)

peak · Accepted Answer · 2018-12-23 03:14:33Z

0

Constructing a dictionary object opens the door to an efficient solution. If your jq has INDEX/2, you could use the following invocation:

jq --arg p "${arr[*]}" '
  INDEX($p | split(" ")[]; .) as $dict
  | .foofoo.barbar 
      |= map(select($dict[.bar|tostring] | not))'

If your jq does not have INDEX/2, then now would be an excellent time to upgrade; otherwise, you could snarf its def by googling: jq "def INDEX"

answered Dec 23, 2018 at 3:14

peak

119k21 gold badges185 silver badges218 bronze badges

Collectives™ on Stack Overflow

Modify a JSON array with inputs from a shell array in only a single pass

3 Answers 3

9 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

9 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related