how to extract and modify inner array objects with parent object data in jq

Question

We are tying to format a json similar to this:

[
{"id": 1,
    "type": "A",
    "changes": [ 
        {"id": 12},
        {"id": 13}
    ],
    "wanted_key": "good",
    "unwanted_key": "aaa"
},
{"id": 2,
    "type": "A",
    "unwanted_key": "aaa"
},
{"id": 3,
    "type": "B",
    "changes": [
        {"id": 31},
        {"id": 32}
    ],
    "unwanted_key": "aaa",
    "unwanted_key2": "aaa"
},
{"id": 4,
    "type": "B",
    "unwanted_key3": "aaa"
},
null,
null,
{"id": 7}
]

into something like this:

[
  {
    "id": 1,
    "type": "A",
    "wanted_key": true  # every record must have this key/value
  },
  {
    "id": 12,  # note: this was in the "changes" property of record id 1
    "type": "A",  # type should be the same type than record id 1
    "wanted_key": true
  },
  {
    "id": 13,  # note: this was in the "changes" property of record id 1
    "type": "A",  # type should be the same type than record id 1
    "wanted_key": true
  },
  {
    "id": 2,
    "type": "A",
    "wanted_key": true
  },
  {
    "id": 3,
    "type": "B",
    "wanted_key": true
  },
  {
    "id": 31,  # note: this was in the "changes" property of record id 3
    "type": "B",  # type should be the same type than record id 3
    "wanted_key": true
  },
  {
    "id": 32,  # note: this was in the "changes" property of record id 3
    "type": "B",  # type should be the same type than record id 3
    "wanted_key": true
  },
  {
    "id": 4,
    "type": "B",
    "wanted_key": true
  },
  {
    "id": 7,
    "type": "UNKN",  # records without a type should have this type
    "wanted_key": true
  }
]

So far, I've been able to:

remove null records
obtain the keys we need with their default
give records without a type a default type

What we are missing:

from records having a changes key, create new records with the type of their parent record
join all records in a single array

Unfortunately we are not entirely sure how to proceed... Any help would be appreciated.

So far our jq goes like this: del(..|nulls) | map({id, type: (.type // "UNKN"), wanted_key: (true)}) | del(..|nulls)

Here's our test code:

https://jqplay.org/s/eLAWwP1ha8P

knittl · Accepted Answer · 2022-09-07 21:33:31Z

2

The following should work:

map(select(values))
| map(., .type as $type | (.changes[]? + {$type}))
| map({id, type: (.type // "UNKN"), wanted_key: true})

Only select non-null values
Return the original items followed by their inner changes array (+ outer type)
Extract 3 properties for output

Multiple map calls can usually be combined, so this becomes:

map(
    select(values)
    | ., (.type as $type | (.changes[]? + {$type}))
    | {id, type: (.type // "UNKN"), wanted_key: true}
)

Another option without variables:

map(
    select(values)
    | ., .changes[]? + {type}
    | {id, type: (.type // "UNKN"), wanted_key: true}
)
# or:
map(select(values))
| map(., .changes[]? + {type})
| map({id, type: (.type // "UNKN"), wanted_key: true})

or even with a separate normalization step for the unknown type:

map(select(values))
| map(.type //= "UNKN")
| map(., .changes[]? + {type})
| map({id, type, wanted_key: true})
# condensed to a single line:
map(select(values) | .type //= "UNKN" | ., .changes[]? + {type} | {id, type, wanted_key: true})

Explanation:

Select only non-null values from the array
If type is not set, create the property with value "UNKN"
Produce the original array items, followed by their nested changes elements extended with the parent type
Reshape objects to only contain properties id, type, and wanted_key.

edited Sep 7, 2022 at 21:33

answered Sep 7, 2022 at 19:39

knittl

269k59 gold badges339 silver badges405 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

pmf Over a year ago

In your last (condensed to a single line) version, select(values).type //= "UNKN" only filters out null values for the assignment, and keeps on processing the unfiltered set. Separate those two steps as in select(values) | .type //= "UNKN" in order to be equivalent to the uncondensed solution.

knittl Over a year ago

@pmf ah yes, thanks for noticing! I even had that version initially, but then removed the pipe again (operator/filter precedence can be a bit confusing at times with jq)

ateam Over a year ago

I'm gonna accept your answer because of the explanations, but honestly, yours, pmf and Inian are all great, thanks to all.

pmf · Accepted Answer · 2022-09-07 19:32:49Z

1

Here's one way:

map(
  select(values)
  | (.type // "UNKN") as $type
  | ., .changes[]?
  | {id, $type, wanted_key: true}
)

[
  {
    "id": 1,
    "type": "A",
    "wanted_key": true
  },
  {
    "id": 12,
    "type": "A",
    "wanted_key": true
  },
  {
    "id": 13,
    "type": "A",
    "wanted_key": true
  },
  {
    "id": 2,
    "type": "A",
    "wanted_key": true
  },
  {
    "id": 3,
    "type": "B",
    "wanted_key": true
  },
  {
    "id": 31,
    "type": "B",
    "wanted_key": true
  },
  {
    "id": 32,
    "type": "B",
    "wanted_key": true
  },
  {
    "id": 4,
    "type": "B",
    "wanted_key": true
  },
  {
    "id": 7,
    "type": "UNKN",
    "wanted_key": true
  }
]

Demo

answered Sep 7, 2022 at 19:32

pmf

38.3k3 gold badges31 silver badges41 bronze badges

Comments

Inian · Accepted Answer · 2022-09-07 19:44:58Z

1

Something like below should work

map( 
  select(type == "object") | 
  ( {id}, {id : ( .changes[]? .id )} ) + 
  { type: (.type // "UNKN"), wanted_key: true }
)

jq play - demo

edited Sep 7, 2022 at 19:44

answered Sep 7, 2022 at 19:33

Inian

87k15 gold badges166 silver badges183 bronze badges

Collectives™ on Stack Overflow

how to extract and modify inner array objects with parent object data in jq

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related