Scala Spark - Get String row values from a WrappedArray

Question

I'm writing a Scala script for Spark and I have a "specialArray" as following :

 specialArray = ...
 specialArray.show(6)
 __________________________ console __________________________________

 specialArray: org.apache.spark.sql.DataFrame = [_VALUE: array<string>]
 +--------------+
 |        _VALUE|
 +--------------+
 |    [fullForm]|
 |    [fullForm]|
 |    [fullForm]|
 |    [fullForm]|
 |    [fullForm]|
 |    [fullForm]|
 |    [fullForm]|
 +--------------+
 only showing top 6 rows

But would like to see the content of those "fullForm" sub-arrays, how would you do this, please ?

Thank you very much in advance!

I have already tried to get the first value in this way :

val resultTest = specialArray.map(s => s.toString).toDF().collect()(0)
__________________________ console __________________________________
resultTest: org.apache.spark.sql.Row = [[WrappedArray(fullForm)]]

So I don't know how to deal with that and I haven't found anything "effective" in thdoc: : https://www.scala-lang.org/api/current/scala/collection/mutable/WrappedArray.html.

If you have any ideas or you have some questions to ask me, feel free to leave a message, thanks:).

You just want to see the content inside _VALUE or perform some other operation? — koiralo
– koiralo, Commented Apr 17, 2018 at 12:48
First of all see what there is inside because I actually don't even know the values I have to expect. And then, maybe perform some other operations. — Olscream
– Olscream, Commented Apr 17, 2018 at 14:24
@Shankar Koirala : I thank you very much for all your answers but it didn't help me since I couldn't have time to work on it again in the meantime, but be sure I will come back to it soon and then I will accept it if necesary ;). Have a nice day ! — Olscream
– Olscream, Commented Apr 26, 2018 at 7:59

vindev · Accepted Answer · 2018-04-18 04:38:03Z

1

Here specialArray is a dataframe, So to see the schema of dataframe you use specialArray.printSchema, Which shows the datatypes inside the dataframe.

If you just want to see the data inside the dataframe you can use

specialArray.show(6, false) the parameter false is not to truncate while displaying long values.

Next thing you can do is use select or withColumn to change the WrappedArray to the comma-separated (or any separator) String

import org.apache.spark.sql.functions._
df.select(concat_ws(",", $"_VALUE")).show(false)
df.withColumn("_VALUE", concat_ws(",", $"_VALUE")).show(false)

Hope this help!

edited Apr 18, 2018 at 4:38

vindev

2,2802 gold badges15 silver badges20 bronze badges

answered Apr 17, 2018 at 13:13

koiralo

23.2k6 gold badges57 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

Olscream Over a year ago

Thanks for answering first of all ! Finally I tried your solution but it doen't seem to display what I would like to see. Or maybe the content of the array is lot of Strings where it is just written "fullForm". Indeed, I tried :

Olscream Over a year ago

import org.apache.spark.sql.functions._ specialArray.printSchema specialArray.select(concat_ws(",", $"_VALUE")).show(false) specialArray.withColumn("_VALUE", concat_ws(",", $"_VALUE")).show(false) specialArray.map(s => s.toString).toDF().collect()(0)(0)

Olscream Over a year ago

Here a screenshot of what I did through Spark, it's more clear : link to my image (photobox.co.uk). Sorry for the multiple edits. Would you have other ideas about what I could still try ? Or do you think the content of the array is a simple String 'fullForm' or 'abbreviation' ?

koiralo Over a year ago

can you share the screenshot?

Olscream Over a year ago

Yes, of course, here you have the image : <kbd><img src="i.imgur.com/IkWl1X0.png" width="100" height="100"></kbd>

|

Collectives™ on Stack Overflow

Scala Spark - Get String row values from a WrappedArray

1 Answer 1

9 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Related