Apache Spark Dataframe - Issue with setting up a not-equal join

Question

I have 2 dataframes that I'm doing a multi-column join. The first pair of columns is doing an equal comparison and the second pair is a not-equals comparison. The code looks like this:

val arule_1w = itemLHS
      .join(itemRHS, itemLHS("CUST_ID") === itemRHS("CUST_ID") && itemLHS("LHS") != itemRHS("RHS")

The resulting data still has rows that contains itemLHS("LHS") = itemRHS("RHS"), which it shouldn't, with the not-equal join. It maybe user error as well but all my research tells me that format is correct. All datatypes are string values.

Thanks for your help!

2 revs user6022341 · Accepted Answer · 2016-09-02 15:03:08Z

4

Correct method is =!= not !=.

edited Sep 2, 2016 at 15:03

community wiki

2 revs
user6022341

Sign up to request clarification or add additional context in comments.

3 Comments

Richard Walden Over a year ago

I've tried that and it throws the following error: cannot resolve '((CUST_ID = CUST_ID) && LHS)' due to data type mismatch: differing types in '((CUST_ID = CUST_ID) && LHS)' (boolean and string). It's like it doesn't understand what !== is.

user6022341 Over a year ago

Oh it is a precedence issue. Try =!=.

Emre Over a year ago

why in the world is it than OK with != but ignores it!!

Siddharth Arekar · Accepted Answer · 2017-07-12 10:10:41Z

-1

Used below syntaxt itemLHS("LHS") !== itemRHS("RHS")

answered Jul 12, 2017 at 10:10

Siddharth Arekar

691 silver badge7 bronze badges

Collectives™ on Stack Overflow

Apache Spark Dataframe - Issue with setting up a not-equal join

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related