Flatmap scala [String, String,List[String]]

Question

I have this prbolem, I have an RDD[(String,String, List[String]), and I would like to "flatmap" it to obtain a RDD[(String,String, String)]:

e.g:

val x :RDD[(String,String,  List[String]) = 
RDD[(a,b, list[ "ra", "re", "ri"])]

I would like get:

val result: RDD[(String,String,String)] = 
RDD[(a, b, ra),(a, b, re),(a, b, ri)])]

akuiper · Accepted Answer · 2017-06-06 14:29:16Z

7

Use flatMap:

val rdd = sc.parallelize(Seq(("a", "b", List("ra", "re", "ri"))))
// rdd: org.apache.spark.rdd.RDD[(String, String, List[String])] = ParallelCollectionRDD[7] at parallelize at <console>:28

rdd.flatMap{ case (x, y, z) => z.map((x, y, _)) }.collect
// res23: Array[(String, String, String)] = Array((a,b,ra), (a,b,re), (a,b,ri))

answered Jun 6, 2017 at 14:29

akuiper

216k33 gold badges362 silver badges379 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Anahcolus · Accepted Answer · 2017-06-06 15:23:32Z

0

This is an alternative way of doing it using flatMap again

val rdd  =  sparkContext.parallelize(Seq(("a", "b", List("ra", "re", "ri"))))
rdd.flatMap(array => array._3.map(list => (array._1, array._2, list))).foreach(println)

edited Jun 6, 2017 at 15:23

answered Jun 6, 2017 at 14:37

Anahcolus

42.1k6 gold badges75 silver badges101 bronze badges

2 Comments

GPI Over a year ago

I do not agree, the type of result here would be RDD[List[(String, String, String)]] while the OP asks for RDD[(String, String, String)]

Anahcolus Over a year ago

Yes you are right @GPI let me update my answer :) thanks for letting me know

Collectives™ on Stack Overflow

Flatmap scala [String, String,List[String]]

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related