Spark dataFrame convert columns Datatype from String to Date

Question

I have the following data, with schema

scala> df2.printSchema()
root
 |-- RowID: integer (nullable = true)
 |-- Order Date: string (nullable = true)

scala> df2.show(5)
+-----+----------+
|RowID|Order Date|
+-----+----------+
|    1|   4/10/15|
|   49|   4/10/15|
|   50|   4/10/15|
|   80|   4/10/15|
|   85|   4/10/15|
+-----+----------+

I want to convert the "Order Date" String column to Date data type, and trying the following with no luck, can anyone please suggest a better way to do this?

scala> df2.select(df2.col("RowID"), df2.col("Order Date"), date_format(df2.col("Order Date"), "M/dd/yy")).show(5)
+-----+----------+-------------------------------+
|RowID|Order Date|date_format(Order Date,M/dd/yy)|
+-----+----------+-------------------------------+
|    1|   4/10/15|                           null|
|   49|   4/10/15|                           null|
|   50|   4/10/15|                           null|
|   80|   4/10/15|                           null|
|   85|   4/10/15|                           null|
+-----+----------+-------------------------------+

Lokesh Kumar P · Accepted Answer · 2016-05-23 09:28:14Z

1

Managed to convert to the unix epoch timestamp, I think from here it is straightforward

scala> df.select(df.col("RowID"), df.col("Order Date"), unix_timestamp(df.col("Order Date"), "M/d/yy")).show(5)
+-----+----------+--------------------------------+
|RowID|Order Date|unixtimestamp(Order Date,M/d/yy)|
+-----+----------+--------------------------------+
|    1|   4/10/15|                      1428604200|
|   49|   4/10/15|                      1428604200|
|   50|   4/10/15|                      1428604200|
|   80|   4/10/15|                      1428604200|
|   85|   4/10/15|                      1428604200|
+-----+----------+--------------------------------+

answered May 23, 2016 at 9:28

Lokesh Kumar P

3695 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark dataFrame convert columns Datatype from String to Date

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related