How to convert dataframe value into Map[String,List[String]]?

Question

I want to convert below dataframe into Map[String,List[String]]. I have changed initial dataframe to get Name columns in List format(using collect_list) but I am not able to convert it into Map[String,List[String]].

DataFrame

+---------+-------+
|City     |  Name |
+---------+-------+
|Mumbai   |[A,B]  |
|Pune     |[C,D]  |
|Delhi    |[A,D]  |
+---------+-------+

Expected Output:

Map(Mumbai -> List(A,B), Pune -> List(C,D), Delhi-> List(A,D))

koiralo · Accepted Answer · 2019-07-26 11:34:03Z

1

You can convert to rdd and collect as Map as below

val df = Seq(
  ("Mumbai", List("A", "B")),
  ("Pune", List("C", "D")),
  ("Delhi", List("A", "D"))
).toDF("city", "name")

val map: collection.Map[String, List[String]] =  df.rdd
  .map(row => (row.getAs[String]("city"), row.getAs[List[String]]("name")))
  .collectAsMap()

Hope this helps!

answered Jul 26, 2019 at 11:34

koiralo

23.2k6 gold badges57 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to convert dataframe value into Map[String,List[String]]?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related