How to extract subArray from Array[Array[Int]] column DataFrame

Question

I have a Dataframe like this:

+---------------------------------------------------------------------+
|ARRAY                                                                |
+---------------------------------------------------------------------+
|[WrappedArray(1, 2, 3), WrappedArray(4, 5, 6), WrappedArray(7, 8, 9)]|
+---------------------------------------------------------------------+

I use this code to create it:

case class MySchema(arr: Array[Array[Int]])
val df = sc.parallelize(Seq(
    Array(Array(1,2,3),
          Array(4,5,6),
          Array(7,8,9))))
      .map(x => MySchema(x))
      .toDF("ARRAY")

I would like to get a result like this:

+-----------+
|ARRAY      |                                                          |
+-----------+
|[1, 2, 3]  |
|[4, 5, 6]  |
|[7, 8, 9]  |
+-----------+

Do you have any idea?

I already try to call an udf to do a flatmap(x => x) on my Array line but I get an incorrect result :

+---------------------------+
|ARRAY                      |
+---------------------------+
|[1, 2, 3, 4, 5, 6, 7, 8, 9]|
+---------------------------+

Thank you for your help

user6022341 · Accepted Answer · 2016-12-13 14:10:54Z

4

You can explode:

import org.apache.spark.sql.functions.{col, explode}
df.select(explode(col("array")))

Collectives™ on Stack Overflow

How to extract subArray from Array[Array[Int]] column DataFrame

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related