Converting dataframe columns into list of tuples

Question

I have:

val DF1 = sparkSession.sql("select col1,col2,col3 from table");
val tupleList = DF1.select("col1","col2").rdd.map(r => (r(0),r(1))).collect()

tupleList.foreach(x=> x.productIterator.foreach(println))

But I do not get all the tuples in the output. Where is the issue?

col1 col2
AA  CCC
AA  BBB 
DD  CCC 
AB  BBB 
Others  BBB 
GG  ALL 
EE  ALL 
Others  ALL 
ALL BBB 
NU FFF 
NU  Others 
Others  Others 
C   FFF

The output I get is: CCC AA BBB AA Others AA Others DD ALL Others ALL GG ALL ALL

Before you wonder, check what tuple.productIterator.foreach(println) gives you? — akuiper
– akuiper, Commented Jan 24, 2017 at 15:18

Sai Neelakantam · Accepted Answer · 2017-01-24 18:28:32Z

15

scala> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
scala> val df1 = hiveContext.sql("select id, name from class_db.students")
scala> df1.show()
+----+-------+
|  id|   name|
+----+-------+
|1001|   John|
|1002|Michael|
+----+-------+

scala> df1.select("id", "name").rdd.map(x => (x.get(0), x.get(1))).collect()
res3: Array[(Any, Any)] = Array((1001,John), (1002,Michael))

edited Jan 24, 2017 at 18:28

answered Jan 24, 2017 at 14:57

Sai Neelakantam

9398 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

WestCoastProjects Over a year ago

Maybe add prose "convert the DataFrame to RDD and then map to a tuple".

ashley Over a year ago

I get "invalid syntax" :(

eshirvana · Accepted Answer · 2022-09-16 19:08:21Z

0

To resolve invalid syntax issue in pyspark use

temp = df1.select('id','name').rdd.map(lambda x: (x[0],x[1])).collect()

edited Sep 16, 2022 at 19:08

eshirvana

24.7k3 gold badges28 silver badges43 bronze badges

answered Dec 12, 2020 at 8:35

Mohankumar Sivakumar

412 bronze badges

Collectives™ on Stack Overflow

Converting dataframe columns into list of tuples

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related