Add a null value column in Spark Data Frame using Java

Question

I have a dataframe and want to add a column of type String with null values. How can it be done using Spark Java API.

I used lit functions, but getting error when tried writing the DF and saveAsTable.

Benjamin Zach · Accepted Answer · 2020-10-23 12:11:50Z

5

df.withColumn("col_name", lit(null).cast("string"))

or

import org.apache.spark.sql.types.StringType

df.withColumn("col_name", lit(null).cast(StringType))

Benjamin Zach

1,6412 gold badges21 silver badges43 bronze badges

answered Oct 23, 2020 at 9:41

user14505699

511 silver badge2 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Thank you for year answer; could maybe elaborate a bit more on why this resolves the question?

jdk2588 · Accepted Answer · 2017-09-18 14:50:25Z

4

Was able to solve by using lit function on the column with null value and type cast the column to String type.

df.withColumn(
 "col_name", functions.lit(null)
).withColumn("col_name", 
  df.col("channel_name").cast(DataTypes.StringType)
)

answered Sep 18, 2017 at 14:50

jdk2588

7922 gold badges9 silver badges25 bronze badges