Slice array of structs using column values

Question

I want to use Spark slice function with start and length defined as Column(s).

def slice(x: Column, start: Int, length: Int): Column

x looks like this:

`|-- x: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- a: double (nullable = true)
 |    |    |-- b : double (nullable = true)
 |    |    |-- c: double (nullable = true)
 |    |    |-- d: string (nullable = true)
 |    |    |-- e: double (nullable = true)
 |    |    |-- f: double (nullable = true)
 |    |    |-- g: long (nullable = true)
 |    |    |-- h: double (nullable = true)
 |    |    |-- i: double (nullable = true)
...
`

any idea on how to achieve this ?

Thanks !

array column (my column was created with struct("a","b","c") as "agg1" — Guillaume
– Guillaume, Commented Jan 3, 2019 at 18:32

Raphael Roth · Accepted Answer · 2019-01-03 19:40:08Z

1

You cannot use the built-in DataFrame DSL function slice for this (as it needs constant slice bounds), you can use an UDF for that. If df is your dataframe and you have a from und until column, then you can do:

val mySlice = udf(
  (data:Seq[Row], from:Int, until:Int) => data.slice(from,until),
  df.schema.fields.find(_.name=="x").get.dataType
)

df
  .select(mySlice($"x",$"from",$"until"))
  .show()

Alternatively, you can use the SQL-Expression in Spark SQL:

df
   .select(expr("slice(x,from,until)"))
   .show()

edited Jan 3, 2019 at 19:40

answered Jan 3, 2019 at 19:30

Raphael Roth

27.3k19 gold badges98 silver badges152 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Guillaume Over a year ago

2 questions: 1. what is df.schema.fields.find(_.name=="x").get.dataType (is there anything missing) , 2. Any idea for a DS (instead of DF)

Raphael Roth Over a year ago

@GuillaumeG this is a trick to get the schema of your array, it lets you return Seq[Row] from your UDF, so you don't need to map the struct to a Tuple. 2: With DS it's trivial, just use the slice method of the scala collection API

Guillaume Over a year ago

My from and to are stored as columns. Would you do something like this ? expr("slice(x,from,length)"))

Guillaume Over a year ago

Thanks for changing your comment. Can you fix the typo in the UDF and I will accept it as final

Raphael Roth Over a year ago

.@GuillaumeG I dont see a typo

|

Collectives™ on Stack Overflow

Slice array of structs using column values

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related