pySpark check Dataframe contains in another Dataframe

Question

Assume I have two Dataframes:

DF1: DATA1, DATA1, DATA2, DATA2

DF2: DATA2

I want to exclude all existence of data in DF2 while keeping duplicates in DF1, what should I do?

Expected result: DATA1, DATA1

Rafa · Accepted Answer · 2023-02-07 03:24:36Z

1

Use left anti When you join two DataFrame using Left Anti Join (leftanti), it returns only columns from the left DataFrame for non-matched records.

df3 = df1.join(df2, df1['id']==df2['id'], how='left_anti')

answered Feb 7, 2023 at 3:24

Rafa

5278 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

TommyQu Over a year ago

This is awesome, and what if I want to keep all data in DF1 that matches to DF2 which means expected result as DATA2, DATA2? I tried to switch the order but doesn't seem to work, it gives me empty result stackoverflow.com/questions/64311964/…

Rafa Over a year ago

if u want matching records do inner join

Ram Ghadiyaram · Accepted Answer · 2023-02-07 06:24:56Z

0

df1.except(df2) will give you rows that present in df1 but not in df2

credits: https://sanori.github.io/2019/08/Compare-Two-Tables-in-SQL/

answered Feb 7, 2023 at 6:24

Ram Ghadiyaram

29.4k16 gold badges102 silver badges133 bronze badges

Collectives™ on Stack Overflow

pySpark check Dataframe contains in another Dataframe

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related