How to add null values in an array in spark scala

Question

val data = Array(-999.9,-0.5, -0.3, 0.0, 0.2, 999.9)
 val dataFrame = sqlContext.createDataFrame(data.map(Tuple1.apply)).toDF("features")

I want to introduce the null entry in the above array. I tried below but it didn't work.

val data = Array(-999.9,-0.5, -0.3, 0.0, 0.2, 999.9, null)

akuiper · Accepted Answer · 2017-04-26 14:31:53Z

4

You need to make the array of type Option and the null will be None:

val data = Array(Some(-999.9),Some(-0.5), Some(-0.3), Some(0.0), Some(0.2), Some(999.9),None)
// data: Array[Option[Double]] = Array(Some(-999.9), Some(-0.5), Some(-0.3), Some(0.0), Some(0.2), Some(999.9), None)

val dataFrame = spark.createDataFrame(data.map(Tuple1.apply)).toDF("features")
// dataFrame: org.apache.spark.sql.DataFrame = [features: double]

dataFrame.show    
+--------+
|features|
+--------+
|  -999.9|
|    -0.5|
|    -0.3|
|     0.0|
|     0.2|
|   999.9|
|    null|
+--------+

answered Apr 26, 2017 at 14:31

akuiper

216k33 gold badges362 silver badges379 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

akuiper Over a year ago

Cool. Glad it helps. Good luck!

Collectives™ on Stack Overflow

How to add null values in an array in spark scala

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related