Scala Comparing time columns in dataframe

Question

I have a dataframe with two columns 'time1' and 'time2' with the format : "hh:mm:ss"

I want to create a new column that is the difference between 'time1' and 'time2'.

I have tried this :

 df.withColumn("diff",
                  datediff(
                         to_timestamp($"time1", "hh:mm:ss"),
                         to_timestamp($"time2", "hh:mm:ss")
                          )
            )

But it always return 0 for diff. What is the correct way to do this ?

data sample :

time1, time2
05:35:30, 05:35:12
07:30:55, 02:39:10
08:35:30, 09:36:10
04:35:30, 05:33:50

Tom Lous · Accepted Answer · 2018-04-16 08:07:50Z

2

First of all the format should be "HH:mm:ss" Second datediff only returns diff in days

So if you want a diff in minutes:

df.withColumn("diffs", 
     (to_timestamp('time1, "HH:mm:ss").cast("bigint") - to_timestamp('time2, "HH:mm:ss")
         .cast("bigint")) / lit(60))

answered Apr 16, 2018 at 8:07

Tom Lous

2,9093 gold badges30 silver badges53 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Scala Comparing time columns in dataframe

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related