Read Array of String from Spark

Question

I saved an Array[String] to a Parquet file from Spark.

To read it I use:

row.getAs[Array[String]]("result")

But get:

java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [Ljava.lang.String;

Here is the result of printSchema():

root                                                                            
 |-- result: array (nullable = true)
 |    |-- element: string (containsNull = true)

How should the getAs() be modified?

Or row.getAs[scala.collection.mutable.WrappedArray[String]]("result") . @Reactormonk I think you should make it an answer. — zero323
– zero323, Commented Oct 18, 2015 at 23:24
@zero323 Seq and WrappedArray work. Do you see any reason to use one over the other? More interested about performance than usage. — BAR
– BAR, Commented Oct 20, 2015 at 21:47
There is no difference. In both cases you have the same type of object (WrappedArray). — zero323
– zero323, Commented Oct 20, 2015 at 21:55
@zero323 I didn't know Seq allows for mutable data types. Good to know. — Reactormonk
– Reactormonk, Commented Oct 20, 2015 at 21:57

Reactormonk · Accepted Answer · 2015-10-20 16:19:41Z

29

Does row.getAs[Seq[String]]("result") work?

answered Oct 20, 2015 at 16:19

Reactormonk

21.8k16 gold badges78 silver badges128 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

BAR Over a year ago

Seems like Spark could use an implicit for this case. If I get a chance will check it out and submit PR.

placeybordeaux Over a year ago

@BAR did you end up submitting a PR or filing a ticket? I didn't see one off the bat and I just got bit by this with 1.6.1

Collectives™ on Stack Overflow

Read Array of String from Spark

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related