Scala spark: Access value in a struct in an array-typed column? (Or, access member of anonymous struct-typed column)

Question

I have a dataframe that has a column which is an array of structs, like:

+-----+-----+------------------+---+----+
|index|state|entries           |0  |1   |
+-----+-----+------------------+---+----+
|0    |KY   |[[A, 45]]         |45 |null|
|1    |OR   |[[A, 30], [B, 10]]|30 |10  |
+-----+-----+------------------+---+----+

where "Entries" are structs with two fields, "name" and "number". I want to be able to grab one of those inner values at a particular index.

One way I could do this is:

df.withColumn(col("entries").getItem(0), "dumbName").select("dumbName.name")

I want to be able to do this with anonymous columns, though, so it would look more like

col("entries").getItem(0).someMagicFunction("name")

So what is the issue here? Its not clear what do you want to do. — koiralo
– koiralo, Commented Jun 1, 2020 at 18:42
I was trying to find a syntax that would work similarly to the "someMagicFunction" implementation, without requiring adding a named column to the dataframe. If you check the approved answer, it gives an answer. — Edward Peters
– Edward Peters, Commented Jun 1, 2020 at 21:08

werner · Accepted Answer · 2020-06-01 21:01:55Z

5

getItem works as magic function:

df.select(col("entries").getItem(0).getItem("Name")).show()

prints

+---------------+
|entries[0].Name|
+---------------+
|              A|
|              A|
+---------------+

It is also possible to use element_at from the functions object (available since 2.4.0):

df.select(element_at('entries, 1).getItem("Name")).show()

prints

+---------------------------+
|element_at(entries, 1).Name|
+---------------------------+
|                          A|
|                          A|
+---------------------------+

For earlier Spark versions it would be possible to use SQL:

df.createOrReplaceGlobalTempView("df")
spark.sql("select entries[0].name from global_temp.df").show()

answered Jun 1, 2020 at 21:01

werner

15k6 gold badges36 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Scala spark: Access value in a struct in an array-typed column? (Or, access member of anonymous struct-typed column)

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related