0

I have two dataframes the first one has two columns , the second one has 5 fields , i want to compare the first dataframe with two special column from the second .If it exists i make an update else i insert the two column in the two special fields . i am new , i need some help to continue thanks

Here what i did 

package Test

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
object TMP_STRUCTURE extends App {

  System.setProperty("hadoop.home.dir", "C:\\hadoop");
  System.setProperty("spark.sql.warehouse.dir", "file:///C:/spark-warehouse");

  val sparkSession = SparkSession.builder.master("local").appName("spark session example").getOrCreate()
  //connect to table TMP_STRUCTURE oracle
  val spark = sparkSession.sqlContext
  val df = spark.load("jdbc",
    Map("url" -> "jdbc:oracle:thin:IPTECH/IPTECH@//localhost:1521/XE",
      "dbtable" -> "IPTECH.TMP_STRUCTURE"))
  df.printSchema()


  val article_groups = spark.load("jdbc", Map(
    "url" -> "jdbc:postgresql://localhost:5432/gemodb?user=postgres&password=maher",
    "dbtable" -> "article_groups"))

  article_groups.printSchema()

}



root
 |-- CODE: string (nullable = false)
 |-- LIBELLE: string (nullable = false)

root
 |-- id: long (nullable = false)
 |-- is_enabled: boolean (nullable = true)
 |-- code: string (nullable = true)
 |-- name: string (nullable = true)
 |-- number: string (nullable = true)

i want to comapare code and libelle with id and name based on the column code and id

Any help thanks

Improve this question
2
  • you should also add sample input dataframes and expected dataframes so that others can help you efficiently. Commented Aug 25, 2017 at 23:59
  • I just answered a very similar question the other day (stackoverflow.com/questions/45700834/…). Expand that to two columns and you have your answer :-) Commented Aug 28, 2017 at 12:58
Related questions
14
Spark SQL: How to append new row to dataframe table (from another table)
22
pyspark dataframe add a column if it doesn't exist
6
INSERT IF NOT EXISTS ELSE UPDATE in Spark SQL
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.