Spark/Scala - unix_timestamp returning the wrong date?

Question

I have a piece of Spark code that looks like this:

df //existing dataframe
  .withColumn("input_date", lit("20190105"))
  .withColumn("input_date_epoch", unix_timestamp(col("input_date"), "YYYYMMdd"))

Now, when I run a df.describe the data returned shows the input_date_epoch column having all values as 1546128000, which when I run through an epoch converter comes out as 2018-12-30 00:00:00, rather than the expected value of 2019-01-05 00:00:00

Am I doing something wrong here?

BlueSheepToken · Accepted Answer · 2019-07-24 10:39:38Z

1

The pattern is wrong, if you want a year with four digits, use yyyy:

spark.range(5)
  .withColumn("input_date", lit("20190105"))
  .withColumn("input_date_epoch", unix_timestamp(col("input_date"), "yyyyMMdd"))

YYYYY actually refers to weekyear, see the documentation

edited Jul 24, 2019 at 10:39

answered Jul 24, 2019 at 10:36

BlueSheepToken

6,1973 gold badges23 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark/Scala - unix_timestamp returning the wrong date?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related