jq - extract nested JSON into multiple arrays

Question

I have some (redacted) JSON at the bottom of this post, a result of running ec2-describe-instances in AWS. The sample JSON has 2 arrays, however this could be larger. I have used jq before but I am having trouble pulling out values that are deeply nested in the JSON. In particular, I am interested in:

InstanceId ["Reservations": [ "Instances": [ "InstanceId":
DeviceName ["Reservations": [ "Instances": [ "BlockDeviceMappings": [ "DeviceName":
VolumeId ["Reservations": [ "Instances": [ "BlockDeviceMappings": [ "Ebs": { "VolumeId":
TAG1 Value/Key
TAG2 Value/Key

Summarised, I want a JSON array I can iterate over that will give me an InstanceID, tags on that instance and a single volume. You will notice that each instance has 3 volumes attached (could be more, or less). I'd like to be able to iterate over each one individually. The end result I am hoping would look something like this:

[
{
"InstanceId": "i-11111111",
"DeviceName": "/dev/sda1",
"VolumeId": "vol-1111111a",
"TAG1": "VALUE1",
"TAG2": "VALUE2"
},
{
"InstanceId": "i-11111111",
"DeviceName": "xvdf",
"VolumeId": "vol-1111111b",
"TAG1": "VALUE1",
"TAG2": "VALUE2"
},
{
"InstanceId": "i-11111111",
"DeviceName": "xvdg",
"VolumeId": "vol-1111111c",
"TAG1": "VALUE1",
"TAG2": "VALUE2"
},
{
"InstanceId": "i-22222222",
"DeviceName": "/dev/sda1",
"VolumeId": "vol-2222222a",
"TAG1": "VALUE1",
"TAG2": "VALUE2"
},
{
"InstanceId": "i-22222222",
"DeviceName": "/dev/sdb",
"VolumeId": "vol-2222222b",
"TAG1": "VALUE1",
"TAG2": "VALUE2"
},
{
"InstanceId": "i-22222222",
"DeviceName": "/dev/sdc",
"VolumeId": "vol-2222222c",
"TAG1": "VALUE1",
"TAG2": "VALUE2"
}
]

This is similar to another question I have posted here on stackoverflow. I have tried so many different combinations but cannot get this to work. Any help is really appreciated.

Here's the JSON:

{
  "Reservations": [
    {
      "OwnerId": "xx",
      "ReservationId": "xx",
      "Groups": [],
      "Instances": [
        {
          "Monitoring": {
            "State": "xx"
          },
          "PublicDnsName": "",
          "Platform": "xx",
          "State": {
            "Code": xx,
            "Name": "xx"
          },
          "EbsOptimized": xx,
          "LaunchTime": "xx",
          "PrivateIpAddress": "xx",
          "ProductCodes": [],
          "VpcId": "xx",
          "StateTransitionReason": "",
          "InstanceId": "i-11111111",
          "ImageId": "xx",
          "PrivateDnsName": "xx",
          "KeyName": "xx",
          "SecurityGroups": [
            {
              "GroupName": "xx",
              "GroupId": "xx"
            },
            {
              "GroupName": "xx",
              "GroupId": "xx"
            }
          ],
          "ClientToken": "xx",
          "SubnetId": "xx",
          "InstanceType": "xx",
          "NetworkInterfaces": [
            {
              "Status": "xx",
              "MacAddress": "xx",
              "SourceDestCheck": xx,
              "VpcId": "xx",
              "Description": "xx",
              "NetworkInterfaceId": "xx",
              "PrivateIpAddresses": [
                {
                  "PrivateDnsName": "xx",
                  "Primary": xx,
                  "PrivateIpAddress": "xx"
                }
              ],
              "PrivateDnsName": "xx",
              "Attachment": {
                "Status": "xx",
                "DeviceIndex": xx,
                "DeleteOnTermination": xx,
                "AttachmentId": "xx",
                "AttachTime": "xx"
              },
              "Groups": [
                {
                  "GroupName": "xx",
                  "GroupId": "xx"
                },
                {
                  "GroupName": "xx",
                  "GroupId": "xx"
                }
              ],
              "SubnetId": "xx",
              "OwnerId": "xx",
              "PrivateIpAddress": "xx"
            }
          ],
          "SourceDestCheck": xx,
          "Placement": {
            "Tenancy": "xx",
            "GroupName": "xx",
            "AvailabilityZone": "xx"
          },
          "Hypervisor": "xx",
          "BlockDeviceMappings": [
            {
              "DeviceName": "/dev/sda1",
              "Ebs": {
                "Status": "attached",
                "DeleteOnTermination": xx,
                "VolumeId": "vol-1111111a",
                "AttachTime": "xx"
              }
            },
            {
              "DeviceName": "xvdf",
              "Ebs": {
                "Status": "attached",
                "DeleteOnTermination": xx,
                "VolumeId": "vol-1111111b",
                "AttachTime": "xx"
              }
            },
            {
              "DeviceName": "xvdg",
              "Ebs": {
                "Status": "attached",
                "DeleteOnTermination": xx,
                "VolumeId": "vol-11111111c",
                "AttachTime": "xx"
              }
            }
          ],
          "Architecture": "xx",
          "RootDeviceType": "xx",
          "RootDeviceName": "xx",
          "VirtualizationType": "hvm",
          "Tags": [
            {
              "Value": "TAG1",
              "Key": "VALUE1"
            },
            {
              "Value": "TAG2",
              "Key": "VALUE2"
            },
            {
              "Value": "TAG3",
              "Key": "VALUE3"
            },
            {
              "Value": "TAG4",
              "Key": "VALUE4"
            },
            {
              "Value": "TAG5",
              "Key": "VALUE5"
            },
            {
              "Value": "TAG6",
              "Key": "Value6"
            }
          ],
          "AmiLaunchIndex": xx
        }
      ]
    },
    {
      "OwnerId": "xx",
      "ReservationId": "xx",
      "Groups": [],
      "Instances": [
        {
          "Monitoring": {
            "State": "xx"
          },
          "PublicDnsName": "",
          "Platform": "xx",
          "State": {
            "Code": xx,
            "Name": "xx"
          },
          "EbsOptimized": xx,
          "LaunchTime": "xx",
          "PrivateIpAddress": "xx",
          "ProductCodes": [],
          "VpcId": "xx",
          "StateTransitionReason": "",
          "InstanceId": "i-22222222",
          "ImageId": "xx",
          "PrivateDnsName": "xx",
          "KeyName": "xx",
          "SecurityGroups": [
            {
              "GroupName": "xx",
              "GroupId": "xx"
            },
            {
              "GroupName": "xx",
              "GroupId": "xx"
            }
          ],
          "ClientToken": "xx",
          "SubnetId": "xx",
          "InstanceType": "xx",
          "NetworkInterfaces": [
            {
              "Status": "xx",
              "MacAddress": "xx",
              "SourceDestCheck": xx,
              "VpcId": "xx",
              "Description": "xx",
              "NetworkInterfaceId": "xx",
              "PrivateIpAddresses": [
                {
                  "PrivateDnsName": "xx",
                  "Primary": xx,
                  "PrivateIpAddress": "xx"
                }
              ],
              "PrivateDnsName": "xx",
              "Attachment": {
                "Status": "xx",
                "DeviceIndex": xx,
                "DeleteOnTermination": xx,
                "AttachmentId": "xx",
                "AttachTime": "xx"
              },
              "Groups": [
                {
                  "GroupName": "xx",
                  "GroupId": "xx"
                },
                {
                  "GroupName": "xx",
                  "GroupId": "xx"
                }
              ],
              "SubnetId": "xx",
              "OwnerId": "xx",
              "PrivateIpAddress": "xx"
            }
          ],
          "SourceDestCheck": xx,
          "Placement": {
            "Tenancy": "xx",
            "GroupName": "xx",
            "AvailabilityZone": "xx"
          },
          "Hypervisor": "xx",
          "BlockDeviceMappings": [
            {
              "DeviceName": "/dev/sda1",
              "Ebs": {
                "Status": "attached",
                "DeleteOnTermination": xx,
                "VolumeId": "vol-2222222a",
                "AttachTime": "xx"
              }
            },
            {
              "DeviceName": "/dev/sdb",
              "Ebs": {
                "Status": "attached",
                "DeleteOnTermination": xx,
                "VolumeId": "vol-2222222b",
                "AttachTime": "xx"
              }
            },
            {
              "DeviceName": "/dev/sdc",
              "Ebs": {
                "Status": "attached",
                "DeleteOnTermination": xx,
                "VolumeId": "vol-2222222c",
                "AttachTime": "xx"
              }
            }
          ],
          "Architecture": "xx",
          "RootDeviceType": "xx",
          "RootDeviceName": "xx",
          "VirtualizationType": "hvm",
          "Tags": [
            {
              "Value": "TAG1",
              "Key": "VALUE1"
            },
            {
              "Value": "TAG2",
              "Key": "VALUE2"
            },
            {
              "Value": "TAG3",
              "Key": "VALUE3"
            },
            {
              "Value": "TAG4",
              "Key": "VALUE4"
            },
            {
              "Value": "TAG5",
              "Key": "VALUE5"
            },
            {
              "Value": "TAG6",
              "Key": "Value6"
            }
          ],
          "AmiLaunchIndex": xx
        }
      ]
    }
]

Edit: an example of something I have tried (this one is not trying to get the tags for simplicities sake:

jq -r '.Reservations.Instances | map(({ InstanceId } + (.BlockDeviceMappings | add) + (.BlockDeviceMappings.Ebs | Add))| { InstanceId, DeviceName, VolumeId })' <(echo "$json_array_windows")

jq: error: Add/0 is not defined at <top-level>, line 1:
.Reservations.Instances | map(({ InstanceId } + (.BlockDeviceMappings | add) + (.BlockDeviceMappings.Ebs | Add))| { InstanceId, DeviceName, VolumeId })                                      
jq: 1 compile error

peak · Accepted Answer · 2016-04-14 18:51:09Z

4

The following jq filter produces a stream of JSON objects as requested. If you really need a JSON array, then simply wrap the entire expression in square brackets.

.Reservations[] | .Instances[]
| { InstanceId }
  + (.BlockDeviceMappings[]
     | { DeviceName,
         "VolumeId": .Ebs.VolumeId } )
  + (.Tags
     | { "TAG1": ( map( select(.Value == "TAG1"))[] | .Key),
         "TAG2": ( map( select(.Value == "TAG2"))[] | .Key) } )

Here is the output (once the input JSON has been corrected):

{
  "InstanceId": "i-11111111",
  "DeviceName": "/dev/sda1",
  "VolumeId": "vol-1111111a",
  "TAG1": "VALUE1",
  "TAG2": "VALUE2"
}
{
  "InstanceId": "i-11111111",
  "DeviceName": "xvdf",
  "VolumeId": "vol-1111111b",
  "TAG1": "VALUE1",
  "TAG2": "VALUE2"
}
{
  "InstanceId": "i-11111111",
  "DeviceName": "xvdg",
  "VolumeId": "vol-11111111c",
  "TAG1": "VALUE1",
  "TAG2": "VALUE2"
}
{
  "InstanceId": "i-22222222",
  "DeviceName": "/dev/sda1",
  "VolumeId": "vol-2222222a",
  "TAG1": "VALUE1",
  "TAG2": "VALUE2"
}
{
  "InstanceId": "i-22222222",
  "DeviceName": "/dev/sdb",
  "VolumeId": "vol-2222222b",
  "TAG1": "VALUE1",
  "TAG2": "VALUE2"
}
{
  "InstanceId": "i-22222222",
  "DeviceName": "/dev/sdc",
  "VolumeId": "vol-2222222c",
  "TAG1": "VALUE1",
  "TAG2": "VALUE2"
}

edited Apr 14, 2016 at 18:51

answered Apr 14, 2016 at 5:12

peak

119k21 gold badges185 silver badges218 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

user6127511 Over a year ago

Thank you so much! I notice though that all DeviceNames are '/dev/sda1' which appears to be in error.

peak Over a year ago

You said "I'd like to be able to iterate over each one individually". As I indicated, I only showed the first three JSON objects.

user6127511 Over a year ago

Apologies for not wording this correctly. I do need the 'DeviceName' to be correctly mapped to the 'VolumeId'. Do you think this is possible?

peak Over a year ago

I think you'll find the revised answer meets your expectations.

user6127511 Over a year ago

Any way to get the value of a TAG that's not found to just say 'null', instead of not showing anything at all? Happy to open another thread

|

Collectives™ on Stack Overflow

jq - extract nested JSON into multiple arrays

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related