Pyspark join multiple dataframes with sql join

Question

coming from SAS I want to join multiple dataframes in one SQL join in PySpark. In SAS, thats possible, however, i get the sense that in Pyspark it is not. My script looks like this:

A.createOrReplaceTempView("A")
B.createOrReplaceTempView("B")
C.createOrReplaceTempView("C")

D = spark.sql("select a.*, b.VAR_B, C.VAR_C
              from A a    left join B b on a.VAR == b.VAR
                          left join C c on  a.VAR == c.VAR")

Is that possible in PySpark? Thank you!

Assaf Segev · Accepted Answer · 2021-09-16 11:38:54Z

2

In PySpark, joins work in a similar way to SQL.

First define a df, for example

df_a = spark.sql('select * from a)
df_b = spark.sql('select * from b)
df_c = spark.sql('select * from c)

Then you can do the join as following -

df_joined_a = df_a.join(df_b, a['VAR'] == b['VAR'], 'left')\
.select(df_a['*'], df_b['VAR'].alias('b_var'))
df_joined_c = df_joined_a.join(df_c, df_joined_a['VAR'] == c['VAR'], 'left')\
.select(df_joined_a['*'], df.c['VAR'])

More examples are available here - https://sparkbyexamples.com/pyspark/pyspark-join-explained-with-examples/

edited Sep 16, 2021 at 11:38

answered Sep 15, 2021 at 9:21

Assaf Segev

3911 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pyspark join multiple dataframes with sql join

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related