Creating Spark Row from CSV String

Question

I am a new to Spark(Using Scala), I am trying few things in RDD to DF conversion etc. I have a String variable for Example:

val myString = "apple, boy, cat, dog"

How can I convert myString to org.apache.spark.sql.Row

I have tried new things like below, but When I am trying to print the length of created row i am getting 1(ONE) where I shall get 4.

val row = org.apache.spark.sql.Row.apply(myString)

val row1 = org.apache.spark.sql.Row(myString) 

val row2 = org.apache.spark.sql.Row.fromSeq(Seq(myString.split(',')))

Anahcolus · Accepted Answer · 2018-05-14 02:46:49Z

3

the correct way is

org.apache.spark.sql.Row.fromSeq(myString.split(','))
//res0: org.apache.spark.sql.Row = [apple, boy, cat, dog]

where myString.split(',') is an Array[String] and converted implicitly to Seq

and if you want to create a dataframe then

val myString = "apple, boy, cat, dog"

val row2 = sc.parallelize(Seq(org.apache.spark.sql.Row.fromSeq(myString.split(','))))
sqlContext.createDataFrame(row2, StructType(Seq(StructField("name1", StringType, true), StructField("name2", StringType), StructField("name3", StringType), StructField("name4", StringType)))).show(false)

which should give you

+-----+-----+-----+-----+
|name1|name2|name3|name4|
+-----+-----+-----+-----+
|apple| boy | cat | dog |
+-----+-----+-----+-----+

where StructType(Seq(StructField("name1", StringType, true), StructField("name2", StringType), StructField("name3", StringType), StructField("name4", StringType))) is schema creation.

edited May 14, 2018 at 2:46

answered May 14, 2018 at 2:40

Anahcolus

42.1k6 gold badges75 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

D V N Over a year ago

Thanks Rameshorg.apache.spark.sql.Row.fromSeq(myString.split(',')) //res0: org.apache.spark.sql.Row = [apple, boy, cat, dog] works perfectly....

Anahcolus Over a year ago

@DVN if the answer helped you then you should consider accepting it :0

Collectives™ on Stack Overflow

Creating Spark Row from CSV String

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related