difference of two columns in spark dataframe using java

Question

Please let me know how can I find difference between two column of a dataset using Spark in Java.

can you post your code and explain what does not work as you expect? — Daniele
– Daniele, Commented Jun 13, 2018 at 20:41
I was able to resolve it in a regular sql way instead of using when() function. posted by answer below. — Bella Ra
– Bella Ra, Commented Jun 14, 2018 at 16:12

Serge Harnyk · Accepted Answer · 2019-04-11 18:10:41Z

1

You can do it next way. Lets assume, you want additional column with difference:

Dataset<Row> ds = // your dataset
ds.withColumn("diff_col", new Column("col1").$minus(new Column("col2"));

answered Apr 11, 2019 at 18:10

Serge Harnyk

1,34911 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

philantrovert · Accepted Answer · 2018-06-14 07:43:24Z

0

Resolved the issue by using sql way of subtracting the two column values.

sqlContext.sql("Select col1, sum(col2)-sum(col3) as resCol from temp group by col1")

philantrovert

10.1k3 gold badges42 silver badges65 bronze badges

answered Jun 13, 2018 at 19:32

Bella Ra

113 bronze badges