Jq iterating array objects to create new json objects

Question

I've been thinking and searching for a long time, but I didn't find out what I'm looking for. I'm using JQ to parse tshark (-ek) json output, but I'm a jq newby When a frame is multivalue I have a JSON similar to this:

 {
      "timestamp": "1525627021656",
      "layers": {
        "frame_time_epoch": [
          "1525627021.656417000"
        ],
        "ip_src": [
          "10.10.10.10"
        ],
        "ip_src_host": [
          "test"
        ],
        "ip_dst": [
          "10.10.10.11"
        ],
        "ip_dst_host": [
          "dest_test"
        ],
        "diameter_Event-Timestamp": [
          "May  6, 2018 19:17:02.000000000 CEST",
          "May  6, 2018 19:17:02.000000000 CEST"
        ],
        "diameter_Origin-Host": [
          "TESTHOST",
          "TESTHOST"
        ],
        "diameter_Destination-Host": [
          "DESTHOST",
          "DESTHOST"
        ],
        "diameter_CC-Request-Type": [
          "2",
          "2"
        ],
        "diameter_CC-Request-Number": [
          "10",
          "3"
        ],
        "diameter_Rating-Group": [
          "9004",
          "9001"
        ],
        "diameter_Called-Station-Id": [
          "testing",
          "testing"
        ],
        "diameter_User-Name": [
          "testuser",
          "testuser"
        ],
        "diameter_Subscription-Id-Data": [
          "66666666666",
          "77777777777"
        ],
        "gtp_qos_version": [
          "0x00000008",
          "0x00000005"
        ],
        "gtp_qos_max_dl": [
          "8640",
          "42"
        ],
        "diameter_Session-Id": [
          "test1;sessionID1;test1",
          "test2;sessionID2;test2"
        ]
      }
    }

As you can see, many keys are array and I want to iterate them to create different json objects in a result like this:

{
    "frame_time_epoch": [
      "1525627021.656417000"
    ],
    "ip_src": [
      "10.10.10.10"
    ],
    "ip_src_host": [
      "test"
    ],
    "ip_dst": [
      "10.10.10.11"
    ],
    "ip_dst_host": [
      "dest_test"
    ],
    "diameter_Event-Timestamp": [
      "May  6, 2018 19:17:02.000000000 CEST"
    ],
    "diameter_Origin-Host": [
      "TESTHOST"
    ],
    "diameter_Destination-Host": [
      "DESTHOST"
    ],
    "diameter_CC-Request-Type": [
      "2"
    ],
    "diameter_CC-Request-Number": [
      "3"
    ],
    "diameter_Rating-Group": [
      "9001"
    ],
    "diameter_Called-Station-Id": [
      "testing"
    ],
    "diameter_User-Name": [
      "testuser"
    ],
    "diameter_Subscription-Id-Data": [
      "77777777777"
    ],
    "gtp_qos_version": [
      "0x00000005"
    ],
    "gtp_qos_max_dl": [
      "42"
    ],
    "diameter_Session-Id": [
      "test2;sessionID2;test2"
    ]
  }
 {
    "frame_time_epoch": [
      "1525627021.656417000"
    ],
    "ip_src": [
      "10.10.10.10"
    ],
    "ip_src_host": [
      "test"
    ],
    "ip_dst": [
      "10.10.10.11"
    ],
    "ip_dst_host": [
      "dest_test"
    ],
    "diameter_Event-Timestamp": [
      "May  6, 2018 19:17:02.000000000 CEST"
    ],
    "diameter_Origin-Host": [
      "TESTHOST"
    ],
    "diameter_Destination-Host": [
      "DESTHOST"
    ],
    "diameter_CC-Request-Type": [
      "2"
    ],
    "diameter_CC-Request-Number": [
      "10"
    ],
    "diameter_Rating-Group": [
      "9004"
    ],
    "diameter_Called-Station-Id": [
      "testing"
    ],
    "diameter_User-Name": [
      "testuser"
    ],
    "diameter_Subscription-Id-Data": [
      "66666666666"
    ],
    "gtp_qos_version": [
      "0x00000008"
    ],
    "gtp_qos_max_dl": [
      "8640"
    ],
    "diameter_Session-Id": [
      "test1;sessionID1;test1"
    ]
  }

Another hand made example: INPUT:

{
    "key_single": ["single_value"],
    "key2": ["single_value"],
    "multiple_value_key": ["value1" , "value2"],
    "any_key_name": ["value4" ,"value5"]
}
{
    "key_single": ["single_value"],
    "key2": ["single_value"],
    "multiple_value_key": ["value6" , "value7", "value8"],
    "any_key_name": ["value9" ,"value10" , "value11"]
}

Desired output:

{
    "key_single": ["single_value"],
    "key2": ["single_value"],
    "multiple_value_key": ["value1"],
    "any_key_name": ["value4"],
}
{
    "key_single": ["single_value"],
    "key2": ["single_value"],
    "multiple_value_key": ["value2"],
    "any_key_name": ["value5"],
}
{
    "key_single": ["single_value"],
    "key2": ["single_value"],
    "multiple_value_key": ["value6"],
    "any_key_name": ["value9"],
}
{
    "key_single": ["single_value"],
    "key2": ["single_value"],
    "multiple_value_key": ["value7"],
    "any_key_name": ["value10"],
}
{
    "key_single": ["single_value"],
    "key2": ["single_value"],
    "multiple_value_key": ["value8"],
    "any_key_name": ["value11"],
}

Could you help Me?

Thanks in advance.

@Roberto - In most cases, it looks like you want to eliminate the duplicates, but you will have to be more explicit, e.g. about "diameter_CC-Request-Number". Also, please fix your sample input to make it valid JSON (there is a superfluous comma). — peak
– peak, Commented May 9, 2018 at 8:24
@RomanPerekhrest, Thanks for your answer but I dont understand your cuestion. :( — Roberto
– Roberto, Commented May 9, 2018 at 9:17
Thanks @peak . I try to be more explicit adding an example in the post. I don't want only to eliminate the duplicates, I need to create single valued json from multivalues arrays in the input too. — Roberto
– Roberto, Commented May 9, 2018 at 9:29

peak · Accepted Answer · 2018-05-09 16:57:31Z

1

It looks like you want to take, in turn, the i-th element of the selected arrays. Using your second example, this could be done like so:

range(0; .multiple_value_key|length) as $i
| . + { multiple_value_key: [.multiple_value_key[$i]],
        any_key_name:       [.any_key_name[$i]] }

The output in compact form:

{"key_single":["single_value"],"key2":["single_value"],"multiple_value_key":["value1"],"any_key_name":["value4"]}
{"key_single":["single_value"],"key2":["single_value"],"multiple_value_key":["value2"],"any_key_name":["value5"]}
{"key_single":["single_value"],"key2":["single_value"],"multiple_value_key":["value6"],"any_key_name":["value9"]}
{"key_single":["single_value"],"key2":["single_value"],"multiple_value_key":["value7"],"any_key_name":["value10"]}
{"key_single":["single_value"],"key2":["single_value"],"multiple_value_key":["value8"],"any_key_name":["value11"]}

answered May 9, 2018 at 16:57

peak

119k21 gold badges185 silver badges218 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Roberto Over a year ago

Thanks @peak!!! so grateful for your answer! But i don't know the names of the fields (any_key_name), so i think i must make another array like: jq -c 'keys in $k, range(0; ."multiple_value_key"|length) as $i | . + { $k: [(.$k)[$i]]}' But It's not working, sorry but I'm totally newby at jq :(

peak Over a year ago

But in your first example, the array-valued keys have values that are arrays of varying lengths. I'm afraid you'll have to be more precise about what you are trying to do.

Roberto Over a year ago

thanks @peak. Yes, there are different length arrays, two types: simples with one element ,and multiples with 2 , 3 or 4. But always the same length every object. These json are from the output of tshark, a packet analyzer. When you capture a frame with one IP packet but inside there are more flows in the application protocol, you have the json I've attached you. With one source and destination IP but maybe two or more subscriber in the diameter layer. In this scenario, I would like to split the frames into single subscriber frames with the IP information too. Have you got it?

peak · Accepted Answer · 2018-05-12 22:40:56Z

Here is a simple solution to the problem as described in the "comments", though the output differs slightly from that shown in the Q.

For clarity, a helper function is defined for producing the $i-th slice of an object, that is, for all array-valued keys with array-length greater than 1, the value is replaced by the $i-th item in the array.

def slice($i):
    map_values(if (type == "array" and length>1)
               then [.[$i]]
               else . end);

The solution is then simply:

.layers
| range(0;  [.[] | length] | max) as $i
| slice($i)

Output

{
  "frame_time_epoch": [
    "1525627021.656417000"
  ],
  "ip_src": [
    "10.10.10.10"
  ],
  "ip_src_host": [
    "test"
  ],
  "ip_dst": [
    "10.10.10.11"
  ],
  "ip_dst_host": [
    "dest_test"
  ],
  "diameter_Event-Timestamp": [
    "May  6, 2018 19:17:02.000000000 CEST"
  ],
  "diameter_Origin-Host": [
    "TESTHOST"
  ],
  "diameter_Destination-Host": [
    "DESTHOST"
  ],
  "diameter_CC-Request-Type": [
    "2"
  ],
  "diameter_CC-Request-Number": [
    "10"
  ],
  "diameter_Rating-Group": [
    "9004"
  ],
  "diameter_Called-Station-Id": [
    "testing"
  ],
  "diameter_User-Name": [
    "testuser"
  ],
  "diameter_Subscription-Id-Data": [
    "66666666666"
  ],
  "gtp_qos_version": [
    "0x00000008"
  ],
  "gtp_qos_max_dl": [
    "8640"
  ],
  "diameter_Session-Id": [
    "test1;sessionID1;test1"
  ]
}
{
  "frame_time_epoch": [
    "1525627021.656417000"
  ],
  "ip_src": [
    "10.10.10.10"
  ],
  "ip_src_host": [
    "test"
  ],
  "ip_dst": [
    "10.10.10.11"
  ],
  "ip_dst_host": [
    "dest_test"
  ],
  "diameter_Event-Timestamp": [
    "May  6, 2018 19:17:02.000000000 CEST"
  ],
  "diameter_Origin-Host": [
    "TESTHOST"
  ],
  "diameter_Destination-Host": [
    "DESTHOST"
  ],
  "diameter_CC-Request-Type": [
    "2"
  ],
  "diameter_CC-Request-Number": [
    "3"
  ],
  "diameter_Rating-Group": [
    "9001"
  ],
  "diameter_Called-Station-Id": [
    "testing"
  ],
  "diameter_User-Name": [
    "testuser"
  ],
  "diameter_Subscription-Id-Data": [
    "77777777777"
  ],
  "gtp_qos_version": [
    "0x00000005"
  ],
  "gtp_qos_max_dl": [
    "42"
  ],
  "diameter_Session-Id": [
    "test2;sessionID2;test2"
  ]
}

Thank you @Peak! I'm testing your last solution and it seem to work! I'll keep you posted to know if finally you got it.
Finally I've checked that your solution fits perfect. Thank you very much Peak!

Collectives™ on Stack Overflow

Jq iterating array objects to create new json objects

2 Answers 2

3 Comments

Output

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Output

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related