dynamically join two spark-scala dataframes on multiple columns without hardcoding join conditions

Question

I would like to join two spark-scala dataframes on multiple columns dynamically. I would to avoid hard coding column name comparison as shown in the following statments;

val joinRes = df1.join(df2, df1("col1") == df2("col1") and df1("col2") == df2("col2"))

The solution for this query already exists in pyspark version --provided in the following link PySpark DataFrame - Join on multiple columns dynamically

I would like to code the same code using spark-scala

Stefan Repcek · Accepted Answer · 2017-04-08 18:18:27Z

9

In scala you do it in similar way like in python but you need to use map and reduce functions:

val sparkSession = SparkSession.builder().getOrCreate()
import sparkSession.implicits._

val df1 = List("a,b", "b,c", "c,d").toDF("col1","col2")
val df2 = List("1,2", "2,c", "3,4").toDF("col1","col2")

val columnsdf1 = df1.columns
val columnsdf2 = df2.columns

val joinExprs = columnsdf1
   .zip(columnsdf2)
   .map{case (c1, c2) => df1(c1) === df2(c2)}
   .reduce(_ && _)

val dfJoinRes = df1.join(df2,joinExprs)

answered Apr 8, 2017 at 18:18

Stefan Repcek

2,6034 gold badges23 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Anhata Over a year ago

val dfJoinRes = df1.join(df2,df1.columns.toSet.intersect(df2.columns.toSet).toSeq, "left") //This code work as well in my case

Stefan Repcek Over a year ago

yes it works, I wanted to post that answer first but I think it would not be complete because what if columns in df1 and df2 has different names?

Collectives™ on Stack Overflow

dynamically join two spark-scala dataframes on multiple columns without hardcoding join conditions

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related