updating a map column in dataframe spark/scala

Question

I have a dataframe origMap with a column that is of type map. I want to add more entries to that map

I'm doing following that's working:

val origMap = df("mapping")

val tempMap = tempDFFields.flatMap(tempField => Array(lit(tempField), tempDF(tempField))): _*)

origMap.withColumn("mapping", tempMap.union(origMap))

tempDFFields is list of column names in tempDF.

I'm creating a map of all colname->colvalue from tempDF and want to add it to original DF. It complains that I'm passing array of Column instead of single instance of Column. how can I pass single instance of column here .. I just want to update the map and store it back.

Example:

Input

origDF

+--------+-----------------------------
|id  | amount       | mapping         | 
|1   | 10           | {a=b, c=d}      |
|3   | 90           | {e=f, g=h}           |

tempDF

+-----
|Id |
|1  |

output: origDF

+--------+-----------------------------
|id  | amount       | mapping          | 
|1   | 10           | {a=b, c=d, id=1} |
|3   | 90           | {e=f, g=h, id=1} |

always with one column? what if tempDF Id contains value 1, 2 and 3. — koiralo
– koiralo, Commented May 18, 2018 at 1:02

koiralo · Accepted Answer · 2018-05-18 01:31:36Z

2

You can create an udf to merge the map as below

val origDF = Seq(
  (1, 10, Map("a" -> "b", "c" -> "d")),
  (3, 90, Map("e" -> "f", "g" -> "h"))
).toDF("id", "amount", "mapping")

If you have a single row DF you can create a map directly
val tmpDF = Map("id" -> "1")

//UDf to merge the two map 
val addToMap = udf((mapping: Map[String, String]) => {mapping ++ tmpDF})

//Use the udf 
origDF.withColumn("mapping", addToMap($"mapping"))

.show(false)

Output:

+---+------+----------------------------+
|id |amount|mapping                     |
+---+------+----------------------------+
|1  |10    |Map(a -> b, c -> d, id -> 1)|
|3  |90    |Map(e -> f, g -> h, id -> 1)|
+---+------+----------------------------+

Hope this helps!

answered May 18, 2018 at 1:31

koiralo

23.2k6 gold badges57 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

updating a map column in dataframe spark/scala

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related