Flatten RDD[(String,Map[String,Int])] to RDD[String,String,Int]

Question

I am trying to flatten an RDD[(String,Map[String,Int])] to RDD[String,String,Int] and ultimately save it as a dataframe.

    val rdd=hashedContent.map(f=>(f._1,f._2.flatMap(x=> (x._1, x._2))))
    val rdd=hashedContent.map(f=>(f._1,f._2.flatMap(x=>x)))

All having type mismatch errors. Any help on how to flatten structures like this one? EDIT:

    hashedContent -- ("A", Map("acs"->2, "sdv"->2, "sfd"->1)),
                     ("B", Map("ass"->2, "fvv"->2, "ffd"->1)),
                      ("c", Map("dg"->2, "vd"->2, "dgr"->1))

please provide the test data and the value for hashedcontent — Chandan Ray
– Chandan Ray, Commented Aug 29, 2018 at 17:09

mtoto · Accepted Answer · 2018-08-29 17:26:18Z

4

You were close:

rdd.flatMap(x => x._2.map(y => (x._1, y._1, y._2)))
   .toDF()
   .show()
+---+---+---+
| _1| _2| _3|
+---+---+---+
|  A|acs|  2|
|  A|sdv|  2|
|  A|sfd|  1|
|  B|ass|  2|
|  B|fvv|  2|
|  B|ffd|  1|
|  c| dg|  2|
|  c| vd|  2|
|  c|dgr|  1|
+---+---+---+

Data

val data = Seq(("A", Map("acs"->2, "sdv"->2, "sfd"->1)),
               ("B", Map("ass"->2, "fvv"->2, "ffd"->1)),
               ("c", Map("dg"->2, "vd"->2, "dgr"->1)))

val rdd = sc.parallelize(data)

answered Aug 29, 2018 at 17:26

mtoto

24.3k4 gold badges62 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Rasika Over a year ago

I am getting error at .toDF() Cannot resolve symbol.

mtoto Over a year ago

try running import sqlContext.implicits._ first, what Spark version you are on?

Tzach Zohar · Accepted Answer · 2018-08-29 17:59:34Z

2

For completeness: an alternative solution (which might be considered more readable) would be to first convert the RDD into a DataFrame, and then to transform its structure using explode:

import org.apache.spark.sql.functions._
import spark.implicits._

rdd.toDF("c1", "map")
  .select($"c1", explode($"map"))
  .show(false)

// same result:
// +---+---+-----+
// |c1 |key|value|
// +---+---+-----+
// |A  |acs|2    |
// |A  |sdv|2    |
// |A  |sfd|1    |
// |B  |ass|2    |
// |B  |fvv|2    |
// |B  |ffd|1    |
// |c  |dg |2    |
// |c  |vd |2    |
// |c  |dgr|1    |
// +---+---+-----+

answered Aug 29, 2018 at 17:59

Tzach Zohar

37.9k3 gold badges83 silver badges86 bronze badges

Collectives™ on Stack Overflow

Flatten RDD[(String,Map[String,Int])] to RDD[String,String,Int]

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related