Spark: Extract Values from Output RDD

Question

I am new in Spark programming. I am trying to extract values from RDD as I got the below output from RDD

(CBI10006,(Some(Himanshu Vasani),None))
(CBI10004,(Some(Sonam Petro),Some(8500)))
(CBI10003,(None,Some(3000)))

And I want to extract above value to below one

(CBI10006,Himanshu Vasani,'')
(CBI10004,Sonam Petro,8500)
(CBI10003,'',3000)

And I have tried FlatMap approch as below

joined.flatMap{case(f1,f2) => (f1,(f2._1,f2._2))} but getting a below error

type mismatch;
 found   : (String, (Option[String], Option[String]))
 required: TraversableOnce[?]
    **joined.flatMap{case(f1,f2) => (f1,(f2._1,f2._2))}**

In your case a map would work, unless you want a RDD[List[String]] as a result — Emiliano Martinez
– Emiliano Martinez, Commented Jan 4, 2022 at 10:59
@DhrumilShah no, I mean that you could just call values on joined to get the answer you want. — Luis Miguel Mejía Suárez
– Luis Miguel Mejía Suárez, Commented Jan 5, 2022 at 3:15

Gabio · Accepted Answer · 2022-01-04 18:37:35Z

2

Using map():

val data = Seq(("CBI10006", (Some("Himanshu Vasani"), None)), ("CBI10004", (Some("Sonam Petro"), Some(8500))),
  ("CBI10003", (None, Some(3000))))
    
spark.sparkContext
  .parallelize(data)
  .map { case (x, y) => (x, y._1.getOrElse(""), y._2.getOrElse("")) }
  .foreach(println)

// output: 
// (CBI10006,Himanshu Vasani,)
// (CBI10004,Sonam Petro,8500)
// (CBI10003,,3000)

answered Jan 4, 2022 at 18:37

Gabio

9,5643 gold badges17 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Dhrumil Shah Over a year ago

Thanks @Gabib it's working....but just for understanding could you suggest me with flatMap option too?

Luis Miguel Mejía Suárez Over a year ago

@DhrumilShah there is no point in using flatMap for a one to one mapping. Why do you want to use that?

Collectives™ on Stack Overflow

Spark: Extract Values from Output RDD

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related