Convert spark dataframe to Array[String]

Question

Can any tell me how to convert Spark dataframe into Array[String] in scala.

I have used the following.

x =df.select(columns.head, columns.tail: _*).collect()

The above snippet gives me an Array[Row] and not Array[String]

thank you for the response. That does the work.Can you tell me if we use.map { row => row.toString() .mkString(",")} eliminate the bracket "[" and "]" when we print it out. — Bharath
– Bharath, Commented Sep 9, 2017 at 21:17
mkString makes a string from an array... This is all Scala knowledge. How would you make an array a sting in Java or Python? My point is, your question/problem is completely outside of Spark — OneCricketeer
– OneCricketeer, Commented Sep 9, 2017 at 21:49

Sohum Sachdev · Accepted Answer · 2017-09-10 05:51:26Z

14

This should do the trick:

df.select(columns: _*).collect.map(_.toSeq)

answered Sep 10, 2017 at 5:51

Sohum Sachdev

1,4071 gold badge12 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user1485864 Over a year ago

How can we fix that columns is generating a Cannot resolve symbol compilation problem?

loneStar · Accepted Answer · 2018-10-02 19:27:20Z

6

DataFrame to Array[String]

data.collect.map(_.toSeq).flatten

You can also use the following

data.collect.map(row=>row.getString(0))

If you have more columns then it is good to use the last one

 data.rdd.map(row=>row.getString(0)).collect

edited Oct 2, 2018 at 19:27

answered Oct 2, 2018 at 19:17

loneStar

4,04026 silver badges42 bronze badges

1 Comment

Max.Futerman Over a year ago

you can replace .map(.toSeq).flatten to .flatMap(.toSeq)

Areeha · Accepted Answer · 2020-04-01 21:19:02Z

1

If you are planning to read the dataset line by line, then you can use the iterator over the dataset:

 Dataset<Row>csv=session.read().format("csv").option("sep",",").option("inferSchema",true).option("escape, "\"").option("header", true).option("multiline",true).load(users/abc/....);

for(Iterator<Row> iter = csv.toLocalIterator(); iter.hasNext();) {
    String[] item = ((iter.next()).toString().split(",");    
}

answered Apr 1, 2020 at 21:19

Areeha

8337 silver badges11 bronze badges

Comments

Bharath · Accepted Answer · 2017-09-09 21:27:03Z

0

The answer was provided by a user named cricket_007. You can use the following to convert Array[Row] to Array[String] :

x =df.select(columns.head, columns.tail: _*).collect().map { row => row.toString() }

Thanks, Bharath

answered Sep 9, 2017 at 21:27

Bharath

4872 gold badges8 silver badges20 bronze badges

3 Comments

OneCricketeer Over a year ago

collect() on a DataFrame is often not how you would use it. Rather, you show() it

Bharath Over a year ago

Hello cricket_007, I don't think show is useful in this case when you want to assign it a variable.

OneCricketeer Over a year ago

Not my point... Collecting the dataframe or RDD to a Scala datatype becomes a bottleneck on the driver process. If you just want to display the output, you select, then show it without a collection

Collectives™ on Stack Overflow

Convert spark dataframe to Array[String]

4 Answers 4

1 Comment

1 Comment

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

1 Comment

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related