Joining multiple table using loop concept using Scala?

Question

I try to execute following piece of code but end up getting error again and again.

My code is as given below. I am trying to pass key value pair while joining tables but fail to pass value.

val mainDF = Seq(("1","acv","34","a"),("2","fbg","56","b"),("3","rty","78","c"))
  .toDF("id","name","age","DBName")
val deltaDF = Seq(("1","gbj","67","a"),("2","gbj","67","a"),("2","jku","88","b"),("4","jku","88","b"),("5","uuu","12","c"))
  .toDF("id","name","age","DBName")
    
val nameMap = Map("TT" -> "id")
for ((k,i) <- nameMap) {
  val updatedDF1 = mainDF.as("main")
    .join(deltaDF.as("delta"), $"main.$i" === $"delta.$i" &&  $"main.DBName" === $"delta.DBName", "outer")
    .select(mainDF.columns.map(c => coalesce($"delta.$c", $"main.$c") as c): _*)
  println(s"key: $k, value: $i")
}
    
updatedDF1.show()

And the error:

Error : <console>:30: error: not found: value updatedDF1
        updatedDF1.show()

If anyone can help me or suggest different way to do the same.

Error:- Error : <console>:31: error: not found: value updatedDF1 updatedDF1.show() — Deepak Kumar
– Deepak Kumar, Commented May 17, 2021 at 9:54
updateDF is defined inside the loop body, and is not visible outside of it. That is one of several reasons loops are rarely used in scala. There is almost always a better way. I could suggest one in this case, but have trouble understanding what you are trying to do in the first place. What do you want your updatedDF to be after the loop? — Dima
– Dima, Commented May 17, 2021 at 13:40

ELinda · Accepted Answer · 2021-05-17 13:39:31Z

1

As comments suggest, the updatedDF1 declaration becomes out of scope outside of the for block and therefore not in the same lexical scope. Move the show action statement into the for block:

val nameMap = Map("TT" -> "id")
for ((k,i) <- nameMap) {
  val updatedDF1 = mainDF.as("main")
    .join(deltaDF.as("delta"), $"main.$i" === $"delta.$i" &&  $"main.DBName" === $"delta.DBName", "outer")
    .select(mainDF.columns.map(c => coalesce($"delta.$c", $"main.$c") as c): _*)
  println(s"key: $k, value: $i")
  updatedDF1.show() 
}

To understand why:

"A block is delimited by braces { ... }."
"The definitions inside a block are only visible from within the block."

Points taken from https://www.scala-exercises.org/scala_tutorial/lexical_scopes

answered May 17, 2021 at 13:39

ELinda

2,8211 gold badge13 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Joining multiple table using loop concept using Scala?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related