Nifi Jolt nested arrays turn into array of custom objects

Question

Am absolutely stumped trying to take nested arrays from JSON input and turn them into an array of objects with proper Keys and values using Nifi Jolt Transform.

The issue I'm reaching is that I need to manually specify the column names. They are not apparent from the JSON response.

Column names are

{
  "column_names": [
    "icao24",
    "callsign",
    "origin_country",
    "time_position",
    "last_contact",
    "long",
    "lat",
    "baro_altitude",
    "on_ground",
    "velocity",
    "true_track",
    "vertical_rate",
    "sensors",
    "geo_altitude",
    "squawk",
    "spi",
    "position_source"
  ]
}

Example input data:

{
  "time": 1675791934,
  "states": [
    [
      "a57b26",
      "N452SM  ",
      "United States",
      1675791621,
      1675791621,
      -105.1168,
      39.9103,
      null,
      true,
      0,
      90,
      null,
      null,
      null,
      null,
      false,
      0
    ],
    [
      "aa56da",
      "UAL1986 ",
      "United States",
      1675791933,
      1675791933,
      -122.1349,
      41.1152,
      10972.8,
      false,
      235.33,
      2.51,
      0,
      null,
      11049,
      null,
      false,
      0
    ]
  ]
}

Expected output:

[
  {
    "icao24": "a57b26",
    "callsign": "N452SM  ",
    "origin_country": "United States",
    "time_position": 1675791621,
    "last_contact": 1675791621,
    "long": -105.1168,
    "lat": 39.9103,
    "baro_altitude": null,
    "on_ground": true,
    "velocity": 0,
    "true_track": 90,
    "vertical_rate": null,
    "sensors": null,
    "geo_altitude": null,
    "squawk": null,
    "spi": false,
    "position_source": 0
  },
  {...}
]

The result that I'm coming up with from shift Jolt spec is...

[
  {
    "operation": "shift",
    "spec": {
      "states": {
        "*": {
          "*": "[&1]"
        }
      }
    }
  }
]

Result:

[
  [
    "a57b26",
    "N452SM  ",
    "United States",
    1675791621,
    1675791621,
    -105.1168,
    39.9103,
    null,
    true,
    0,
    90,
    null,
    null,
    null,
    null,
    false,
    0
  ],
  [
    "aa56da",
    "UAL1986 ",
    "United States",
    1675791933,
    1675791933,
    -122.1349,
    41.1152,
    10972.8,
    false,
    235.33,
    2.51,
    0,
    null,
    11049,
    null,
    false,
    0
  ]
]

I'd be able to possibly figure it out if the column names were sent in the payload, but unfortunately I need to set them manually in the transformation.

I did happen to find a helpful stack of a similar situation, but again, the solution looks like heavy duty regex and I'm just not that quite well versed in Nifi Jolt yet.

Jolt reference first element in array as target name

Barbaros Özhan · Accepted Answer · 2023-02-08 20:31:50Z

2

Ended up solving this on my own :

[
  {
    "operation": "default",
    "spec": {
      "temp": [
        [
          "icao24",
          "callsign",
          "origin_country",
          "time_position",
          "last_contact",
          "long",
          "lat",
          "baro_altitude",
          "on_ground",
          "velocity",
          "true_track",
          "vertical_rate",
          "sensors",
          "geo_altitude",
          "squawk",
          "spi",
          "position_source"
        ]
      ]
    }
  },
  {
    "operation": "shift",
    "spec": {
      "temp": {
        "*": "states[]"
      },
      "states": {
        "*": "states[]"
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "states": {
        "*": {
          "*": "[&1].@(2,[0].[&])"
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "0": null,
      "*": "[]"
    }
  }
]

And this is the expected output :

[
  {
    "icao24": "a57b26",
    "callsign": "N452SM  ",
    "origin_country": "United States",
    "time_position": 1675791621,
    "last_contact": 1675791621,
    "long": -105.1168,
    "lat": 39.9103,
    "baro_altitude": null,
    "on_ground": true,
    "velocity": 0,
    "true_track": 90,
    "vertical_rate": null,
    "sensors": null,
    "geo_altitude": null,
    "squawk": null,
    "spi": false,
    "position_source": 0
  },
  {
    "icao24": "aa56da",
    "callsign": "UAL1986 ",
    "origin_country": "United States",
    "time_position": 1675791933,
    "last_contact": 1675791933,
    "long": -122.1349,
    "lat": 41.1152,
    "baro_altitude": 10972.8,
    "on_ground": false,
    "velocity": 235.33,
    "true_track": 2.51,
    "vertical_rate": 0,
    "sensors": null,
    "geo_altitude": 11049,
    "squawk": null,
    "spi": false,
    "position_source": 0
  }
]

edited Feb 8, 2023 at 20:31

Barbaros Özhan

65.9k11 gold badges36 silver badges64 bronze badges

answered Feb 8, 2023 at 19:34

james

591 silver badge5 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Scorpy Over a year ago

@Barbaros could you explain a bit what you did in the third operation to stitch them together. I am getting confused with the levels. [&1] gets the value inside states array. ".@(2,[0].[&]" is getting the values from the first array. So when you go two levels up here is it the new array after the [&1] change or before it.

Community Over a year ago

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

Collectives™ on Stack Overflow

Nifi Jolt nested arrays turn into array of custom objects

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related