Extracting value from nested array and struct spark

Question

I have a schema like the below. I was wondering what is the best way in spark to select the elements seat and drive then cast it into a string. I am reading this in a dataframe with spark 1.6.

|-- cars: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- carId: string (nullable = true)
 |    |    |-- carCode: string (nullable = true)
 |    |    |-- carNumber: string (nullable = true)
 |    |    |-- features: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- seat: string (nullable = true)
 |    |    |    |    |-- drive: string (nullable = true)

The output of cars.features as car_features in json:

"cars_features":[[{"seat":"Auto","drive":"Manual"}]]

I am trying to select "Auto" and put it into a dataframe column and "Manual" and put into another column.

current attempt returns the whole structure as:

+-------------------+
|car_features       |
+-------------------+
|  [[Auto,Manual]]  |
+-------------------+

col("car.features").getItem(0).as("car_features_seat")

so you want select seat and drive as arrays of arrays or just as an array or as rows? — Raphael Roth
– Raphael Roth, Commented Sep 3, 2019 at 19:13

Defcon · Accepted Answer · 2019-09-11 15:09:41Z

1

I had to drill into array twice:

col("car.features").getItem(0).getItem(0).getItem("seat").cast("String").as("car_features_seat")

This extracts "Auto"

answered Sep 11, 2019 at 15:09

Defcon

8273 gold badges21 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Extracting value from nested array and struct spark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related