Scala RDD[String] to RDD[String,String]

Question

I have a RDD[String] which contains following data:

data format : ('Movie Name','Actress Name')

('Night of the Demons (2009)  (uncredited)', '"Steff", Stefanie Oxmann Mcgaha')
('The Bad Lieutenant: Port of Call - New Orleans (2009)  (uncredited)', '"Steff", Stefanie Oxmann Mcgaha') 
('"Please Like Me" (2013) {All You Can Eat (#1.4)}', '$haniqua') 
('"Please Like Me" (2013) {French Toast (#1.2)}', '$haniqua') 
('"Please Like Me" (2013) {Horrible Sandwiches (#1.6)}', '$haniqua')

I want to convert this to RDD[String,String] such as first element within ' ' will be my first String in RDD and second element within ' ' will be my second String in RDD.

I tried this:

val rdd1 = sc.textFile("/home/user1/Documents/TestingScala/actress"
val splitRdd = rdd1.map( line => line.split(",") )
splitRdd.foreach(println)

but it's giving me an error as :

[Ljava.lang.String;@7741fb9
[Ljava.lang.String;@225f63a5
[Ljava.lang.String;@63640bc4
[Ljava.lang.String;@1354c1de

That isn't an error message, that's the object-ids for a bunch of strings. — Michael Lorton
– Michael Lorton, Commented Oct 8, 2016 at 1:41
@Malvolio Can you please tell me how can I remove that error — user225508
– user225508, Commented Oct 8, 2016 at 1:50

Community · Accepted Answer · 2017-05-23 12:00:43Z

5

[Ljava.lang.String;@7741fb9 is not an error, This is wt is printed when you try to print an array.

[ - an single-dimensional array

L - the array contains a class or interface

java.lang.String - the type of objects in the array

@ - joins the string together

7741fb9 the hashcode of the object.

To print String array you can try this code:

import scala.runtime.ScalaRunTime._
splitRdd.foreach(array => println(stringOf(array)))

Source

edited May 23, 2017 at 12:00

CommunityBot

11 silver badge

answered Oct 8, 2016 at 6:11

Pawan B

4,6332 gold badges27 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Kris · Accepted Answer · 2016-10-08 17:03:27Z

0

It's not an error. we could also use flatMap() here to avoid confusion,

val rdd1 = sc.textFile("/home/user1/Documents/TestingScala/actress"
rdd1.flatMap( line => line.split(",")).foreach(println)

Here, The input function to map returns a single element (array), while the flatMap returns a list of elements (0 or more). Also, the output of the flatMap is flattened.

edited Oct 8, 2016 at 17:03

answered Oct 8, 2016 at 13:35

Kris

1,7641 gold badge14 silver badges15 bronze badges

Comments

KiranM · Accepted Answer · 2016-10-09 02:09:04Z

0

Since it is csv file with field-enclosed & row-enclosed, you need to read the file using regular expressions. Simple split doesn't work.

answered Oct 9, 2016 at 2:09

KiranM

1,3241 gold badge11 silver badges20 bronze badges

Comments

Shankar · Accepted Answer · 2016-10-15 15:56:34Z

0

Try this to convert RDD[String] to RDD[String,String]

val rdd1 = sc.textFile("/home/user1/Documents/TestingScala/actress"
val splitRdd = rdd1.map( line => (line.split(",")(0), line.split(",")(1)) )

The above line returns the rdd as key, value pair [Tuple] RDD.

answered Oct 15, 2016 at 15:56

Shankar

9,02926 gold badges100 silver badges172 bronze badges

Collectives™ on Stack Overflow

Scala RDD[String] to RDD[String,String]

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related