Convert a Spark DataFrame into a list of objects

Question

I know that it is possible to convert a dataframe column into a list using something like:

dataFrame.select("ColumnName").rdd.map(r => r(0)).collect()

Let's say I already know the schema of the dataframe and correspondingly I created a case class such as :

case class Synonym(URI: String, similarity: Double, FURI: String)

is there an efficient way to get a list of Synonym objects from the data of the dataframe?

In other words, I am trying to create a mapper that would convert each row of the dataframe into an object of my case class and then return this object in a way that I can have a list of these objects at the end of the operation. is this possible in an efficient nice way?

Tzach Zohar · Accepted Answer · 2017-11-02 21:40:22Z

9

Use as[Synonym] to get a Dataset[Synonym] which you can then collect to get an Array[Synonym]:

val result = dataframe.as[Synonym].collect()

answered Nov 2, 2017 at 21:40

Tzach Zohar

37.9k3 gold badges83 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

user8877342 · Accepted Answer · 2017-11-02 21:40:42Z

3

Use typed Dataset:

df.select("URI", "similarity", "FURI").as[Synonym].collect

answered Nov 2, 2017 at 21:40

user8877342

311 bronze badge

Collectives™ on Stack Overflow

Convert a Spark DataFrame into a list of objects

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related