8

Im using Spark 2.0. I have a column of my dataframe containing a WrappedArray of WrappedArrays of Float.

An example of a row would be:

[[1.0 2.0 2.0][6.0 5.0 2.0][4.0 2.0 3.0]]

Im trying to transform this column into an Array[Array[Float]].

What I tried so far is the following:

dataframe.select("mycolumn").rdd.map(r => r.asInstanceOf[Array[Array[Float]]])

but I get the following error:

Caused by: java.lang.ClassCastException:
 org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to [[F

Any idea would be highly appreciated. Thanks

Improve this question

2 Answers 2

Reset to default
4

Try this:

  val wawa: WrappedArray[WrappedArray[Float]] = null 
  val res: Array[Array[Float]] = wawa.map(inner => inner.array).toArray

It compiles for me

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Thanks Sami, your answer led me to the final resolution. I'll update my own answer with the exact code for those like me who started with a dataframe.
My first question at Stackoverflow got 3 answers none of them worked, but by combing them I found a solution. :D
3

Following @sami-badawi 's answer I am posting the answer for those like me who started from a dataframe.

dataframe.select("mycolumn").rdd.map
(row => row.get(0).asInstanceOf[WrappedArray[WrappedArray[Float]]].array.map(x=>x.toArray))
Improve this answer

1 Comment

I tried to print out the values by below: rdd.map(row => row.get(0).asInstanceOf[WrappedArray[WrappedArray[String]]].toSeq.map(x=>x.toSeq.foreach(println)))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.