Spark DataFrame wrap struct<.. into array of struct<

Question

I'm trying to modify a Dataframe which is generated by an external library. I receive a Dataframe with this schema:

root
 |-- child: struct (nullable = true)
 |    |-- child_id: long (nullable = true)

I would like to wrap the child struct above into an Array, as shown in the box below.

root
 |-- child: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- child_id: long (nullable = true)

I tried to define an UDF:

//the two lines below are an example, in real i get the Dataframe from an  external library. 
val seq = sc.parallelize(Seq("""{ "child": { "child_id": 1}}"""))
val df = sqlContext.read.json(seq)

val myUDF = udf((x: Row) => Array(x))
val df2 = df.withColumn("children",myUDF($"child"))

But i get an exception: "Schema for type org.apache.spark.sql.Row is not supported"

I'm working with Spark 2.1.1.

The real DataFrame is very complex, is there a solution that allow modifying the schema without listing the name or the position of the fields in the child table? For the same reason I also would rather not to map to explicit case classes.

Thank you in advance for any help!

Anahcolus · Accepted Answer · 2017-08-04 06:20:01Z

6

You can use array inbuilt function to get your desired result as

import org.apache.spark.sql.functions._
val df2 = df.withColumn("child", array("child"))

this will update the same column, if you want it in separate column then do

import org.apache.spark.sql.functions._
val df2 = df.withColumn("children", array("child"))

answered Aug 4, 2017 at 6:20

Anahcolus

42.1k6 gold badges75 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Anahcolus Over a year ago

No :) I guess you are not. Thanks for the acceptance and upvotes :)

Collectives™ on Stack Overflow

Spark DataFrame wrap struct<.. into array of struct<

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related