Spark dataframe convert string to timestamp - returns null for empty value

Question

I have a spark app, that need to convert from string to timestamp below is my code.

val df = sc.parallelize(Seq("09/18/2017","")).toDF("sDate")
+----------+
|     sDate|
+----------+
|09/18/2017|
|          |
+----------+
val ts = unix_timestamp($"sDate","MM/dd/yyyy").cast("timestamp")
df.withColumn("ts", ts).show()
+----------+--------------------+
|     sDate|                  ts|
+----------+--------------------+
|09/18/2017|2017-09-18 00:00:...|
|          |                null|
+----------+--------------------+

The conversion is doing good, but if the value is empty , I'm getting null after casting.

Is there any way to return empty if the source value is empty.

Anahcolus · Accepted Answer · 2017-09-21 07:15:34Z

1

you can use when function as below

import org.apache.spark.sql.functions._
val ts = unix_timestamp($"sDate","MM/dd/yyyy").cast("timestamp")
df.withColumn("ts", when(ts.isNotNull, ts).otherwise(lit("empty"))).show()

which would give you output as

+----------+-------------------+
|     sDate|                 ts|
+----------+-------------------+
|09/18/2017|2017-09-18 00:00:00|
|          |              empty|
+----------+-------------------+

answered Sep 21, 2017 at 7:15

Anahcolus

42.1k6 gold badges75 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark dataframe convert string to timestamp - returns null for empty value

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related