I have case where i have to combine a String with a RDD out(String)
String 1
date=
String 2 (RDD of String)
20140101
20140102
20140103
....
Output as
date=20140101, date=20140102, date=20140103,.....
Please help me to achieve this result
Add a comment
|
2 Answers
In Spark you would do that like this:
// Some sample data in an RDD
val data = List("20140101", "20140102", "20140103")
val rdd = sc.parallelize(data)
val concatenated = rdd.map(elem => s"date=$elem").reduce(_ + ", " + _)
Be aware thatreduce is an action. Therefore the result of reduce needs to fit in the memory of the driver.
1 Comment
Federico
To avoid the trailing comma:
val concatenated = rdd.map(elem => s"date=$elem").reduce(_ + ", " + _)string2.map(d => s"$string1=$d").mkString(",")
4 Comments
VB_
Thanks for your help, i need it as a combined string. All tuple should be concatenated. this solution doesn't give the concatenated value. Ideally i should do flatmap to translate rows into columns(just for example)
Dani
Sorry, I don't think I understand. Could you please elaborate?
VB_
Hi, I get a RDD[String] of distinct dates and i have a String variable val foo = "date=". I need to concatenate both the outputs and output should be a string i.e. date=20140901,date=20140921,date=20140929... comma separated.
Dani
Right, how about this? val foo = "date" val strings = sc.parallelize(Seq("20140901", "20140921", "20140929")) strings.map(str => s"$foo=$str").reduce(_ + ", " + _) If you have an iterable of type String (instead of an RDD), you could apply the mkString method that I mentioned