Turn an array of objects into an object of arrays with jq

Question

I would like to use jq to turn an array of objects into an object of arrays.

Consider if I have the following two files:

file1.json:

{
  "key1": 5,
  "key2": 10
}

file2.json:

{
  "key1": 2
}

I would like to merge them together to form:

{
  "key1": [5, 2]
  "key2": [10, null]
}

It's easy to do this with one field per jq command, but I can't figure out how to do it with all the fields at once. My thought is that I need to convert all the values to arrays and then use reduce with *, but I couldn't get it to work.

The jq command needs to work for an arbitrary number of files (more than 2).

peak · Accepted Answer · 2015-12-14 22:50:28Z

2

The following will merge an array of objects into a single object as you describe (in particular, null is used as a filler, which certainly makes sense if, for example, each input object is regarded as an "observation"), but please note that merge/0 as defined here makes no assumptions about the sets of keys in the input objects, and could potentially be made much faster depending on the assumptions that could be made.

def merge:
  def allkeys: map(keys) | add | unique;
  allkeys as $allkeys
  | reduce .[] as $in ({};
     reduce $allkeys[] as $k (.;
       . + {($k): (.[$k] + [$in[$k]]) } ));

merge

Now use the "slurp" option, e.g.:

 $ jq -s -f merge.jq file*.json

edited Dec 14, 2015 at 22:50

answered Dec 14, 2015 at 19:18

peak

119k21 gold badges185 silver badges218 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

schmmd Over a year ago

How would you get this to work with an arbitrary number of files?

peak Over a year ago

@schmmd -- Please note that the original version of merge/0 was not sufficiently general.

user3899165 · Accepted Answer · 2015-12-14 20:37:27Z

2

This is probably slower than @peak's solution, but maybe easier to read:

map(to_entries)
| flatten(1)
| group_by(.key)
| map({
    key: .[0].key,
    value: map(.value)})
| from_entries

Use it like:

jq -s 'map(to_entries) | flatten(1) | group_by(.key) | map({key: .[0].key, value: map(.value)}) | from_entries' file*.json

Unlike your desired output, it does not put nulls in place of missing values, which may be useful if "null" is a valid value in itself.

answered Dec 14, 2015 at 20:37

user3899165

Comments

jq170727 · Accepted Answer · 2017-08-26 03:39:58Z

Here is another solution.

  (add | keys) as $all                                                  # get all keys
| map( . as $o | reduce $all[] as $k ({}; .[$k] = $o[$k]) )             # normalize objects
|    ( . as $a | reduce $all[] as $k ({}; .[$k] = ($a | map(.[$k]))) )  # transpose

To run it use the -s option, e.g:

$ jq -s -f filter.jq file1.json file2.json

Sample output:

{
  "key1": [
    5,
    2
  ],
  "key2": [
    10,
    null
  ]
}

If you have more files, e.g. file3.json

{
  "key3": 2
}

just add them to the command line

$ jq -s -f filter.jq file1.json file2.json file3.json

Sample output:

{
  "key1": [
    5,
    2,
    null
  ],
  "key2": [
    10,
    null,
    null
  ],
  "key3": [
    null,
    null,
    2
  ]
}

Collectives™ on Stack Overflow

Turn an array of objects into an object of arrays with jq

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related