Save spark DataFrame to csv file with map<string,string> column type

Question

I have wrote udf function that convert Map[String,String] values to String:

 udf("mapToString", (input: Map[String,String]) => input.mkString(","))

spark-shell give me error:

    <console>:24: error: overloaded method value udf with alternatives:
  (f: AnyRef,dataType: org.apache.spark.sql.types.DataType)org.apache.spark.sql.expressions.UserDefinedFunction <and> 
...
cannot be applied to (String, Map[String,String] => String)
       udf("mapToString", (input: Map[String,String]) => input.mkString(","))

Is any method to convert column of Map[String,String] values to string values? I need this conversion because i need save dataframe as csv file

philantrovert · Accepted Answer · 2017-09-21 12:42:16Z

4

Assuming that you have a DataFrame as

+---+--------------+
|id |map           |
+---+--------------+
|1  |Map(200 -> DS)|
|2  |Map(300 -> CP)|
+---+--------------+

with the following schema

root
 |-- id: integer (nullable = false)
 |-- map: map (nullable = true)
 |    |-- key: string
 |    |-- value: string (valueContainsNull = true)

You can write a udf which looks like :

def mapToString = udf((map: collection.immutable.Map[String, String]) => 
                       map.mkString.replace(" -> ", ","))

and use the udf function with withColumn API as

df.withColumn("map", mapToString($"map"))

you should have final DataFrame where Map is changed to String

+---+------+
|id |map   |
+---+------+
|1  |200,DS|
|2  |300,CP|
+---+------+

root
 |-- id: integer (nullable = false)
 |-- map: string (nullable = true)

edited Sep 21, 2017 at 12:42

philantrovert

10.1k3 gold badges42 silver badges65 bronze badges

answered Sep 21, 2017 at 12:15

Anahcolus

42.1k6 gold badges75 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Save spark DataFrame to csv file with map<string,string> column type

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related