JQ will not deep copy arrays with recursive algorithm

Question

When using this algorithm:

jq -s 'def deepmerge(a;b):
             reduce b[] as $item (a;
               reduce ($item | keys_unsorted[]) as $key (.;
                 $item[$key] as $val | ($val | type) as $type | .[$key] =
                 if ($type == "object") then
                   deepmerge({}; [if .[$key] == null then {} else .[$key] end, $val])
                 elif ($type == "array") then
                   (.[$key] + $val | unique)
                 else
                   $val
                 end)
               );
             deepmerge({}; .)' test1.json test2.json > merged-1.json

nested arrays are not concatenated/merged, but copied sequentially. I assume this is because when the method stack created by the recursive algorithm unwinds, values are written in the order node >> root, and deeper nested child values are thus always overwritten by their higher parents.

Input files:

test1.json

{
  "item1" : "test1",
  "item2" : "test2",
  "nestedItemsArray1" : [{
    "nestedItem1Item1" : "nestItem1Item1",
    "nestedItem1Array1" : ["nestedItem1Array1Item1", "nestedItem1Array1Item2"] 
  },
  {
      "nestedItem1Item1" : "nestItem1Item2",
      "nestedItem1Array1" : ["nestedItem1Array1Item3", "nestedItem1Array1Item4"]
  }]
}

test2.json

{
  "item1" : "test3",
  "item2" : "test4",
  "array1" : [ "array1item3", "array1item4" ],
  "nestedItemsArray1" : [{
    "nestedItem1Item1" : "nestItem1Item1",
    "nestedItem1Array1" : ["nestedItem1Array1Item5", "nestedItem1Array1Item6"]
  },
  {
      "nestedItem1Item1" : "nestItem1Item2",
      "nestedItem1Array1" : ["nestedItem1Array1Item7", "nestedItem1Array1Item8"]
  }]
}

Actual result:

{
  "item1": "test3",
  "item2": "test4",
  "nestedItemsArray1": [
    {
      "nestedItem1Item1": "nestItem1Item1",
      "nestedItem1Array1": [
        "nestedItem1Array1Item1",
        "nestedItem1Array1Item2"
      ]
    },
    {
      "nestedItem1Item1": "nestItem1Item2",
      "nestedItem1Array1": [
        "nestedItem1Array1Item3",
        "nestedItem1Array1Item4"
      ]
    },
    {
      "nestedItem1Item1": "nestItem1Item1",
      "nestedItem1Array1": [
        "nestedItem1Array1Item5",
        "nestedItem1Array1Item6"
      ]
    },
    {
      "nestedItem1Item1": "nestItem1Item2",
      "nestedItem1Array1": [
        "nestedItem1Array1Item7",
        "nestedItem1Array1Item8"
      ]
    }
  ],
  "array1": [
    "array1item3",
    "array1item4"
  ]
}

Expected result:

{
  "item1": "test3",
  "item2": "test4",
  "nestedItemsArray1": [
    {
      "nestedItem1Item1": "nestItem1Item1",
      "nestedItem1Array1": [
        "nestedItem1Array1Item1",
        "nestedItem1Array1Item2",
        "nestedItem1Array1Item5",
        "nestedItem1Array1Item6"
      ]
    },
    {
      "nestedItem1Item1": "nestItem1Item2",
      "nestedItem1Array1": [
        "nestedItem1Array1Item3",
        "nestedItem1Array1Item4",
        "nestedItem1Array1Item7",
        "nestedItem1Array1Item8"
      ]
    }
  ],
  "array1": [
    "array1item3",
    "array1item4"
  ]
}

Of course this is just one array deep, but I would like this to work on any number of nested levels, regardless of values being nested inside arrays or objects.

pmf · Accepted Answer · 2025-09-18 14:37:17Z

There is no canonical way of merging. jq implements one (basic) interpretation by simply overwriting matching paths, which includes array indices (which start again at zero for a colliding array). The way I understand your expected output is that you want to extend array indices if the array is the parent of a scalar, while relying on the builtin merger otherwise.

To achieve this, I tried to rebuild the way jq merges as jq --stream -n 'reduce (inputs | select(has(1))) as [$p,$v] (.; setpath($p;$v)), then extended the path's last item to a colliding array's length if the original index doesn't exceed it by itself:

jq --stream -n '
  reduce (inputs | select(has(1))) as [$p,$v] (.; setpath(
    if $p | last | type == "number"
    then $p[:-1] + [fmax($p | last; getpath($p[:-1]) | length)]
    else $p end;
    $v
  ))
' file1.json file2.json

{
  "item1": "test3",
  "item2": "test4",
  "nestedItemsArray1": [
    {
      "nestedItem1Item1": "nestItem1Item1",
      "nestedItem1Array1": [
        "nestedItem1Array1Item1",
        "nestedItem1Array1Item2",
        "nestedItem1Array1Item5",
        "nestedItem1Array1Item6"
      ]
    },
    {
      "nestedItem1Item1": "nestItem1Item2",
      "nestedItem1Array1": [
        "nestedItem1Array1Item3",
        "nestedItem1Array1Item4",
        "nestedItem1Array1Item7",
        "nestedItem1Array1Item8"
      ]
    }
  ],
  "array1": [
    "array1item3",
    "array1item4"
  ]
}

Note: If the command-line argument --stream comes in your way with your other processing, you can also move it inside your custom function by using the tostream builtin instead.

def deepmerge(in): reduce (in | tostream | select(has(1))) as [$p,$v] (.;
  setpath(
    if $p | last | type == "number"
    then $p[:-1] + [fmax($p | last; getpath($p[:-1]) | length)]
    else $p end;
    $v
  )
);

deepmerge(inputs)

Demo

Collectives™ on Stack Overflow

JQ will not deep copy arrays with recursive algorithm

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related