Scala - Encoder missing for type stored in dataset

Question

I am trying to run the following command in Scala 2.2

   val x_test0 = cn_train.map( { case row => row.toSeq.toArray } )

And I keep getting the following mistake

 error: Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.

I have already imported implicits._ through the following commands:

val spark = SparkSession.builder.getOrCreate()
import spark.implicits._

which line is it exactly? this error is shown when you try to create dataset without encoders defined. — Anahcolus
– Anahcolus, Commented Mar 6, 2018 at 3:32
The error shows like this: error: Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases. val x_test0 = cn_train.map( { case row => row.toSeq.toArray } ) — rmj
– rmj, Commented Mar 6, 2018 at 3:40
The error message tells you that you can't force an array into a dataframe. Try cn_train.rdd.map{ row => row.toSeq.toArray }, this would at least give you an RDD of arrays. Would that be sufficient? — Andrey Tyukin
– Andrey Tyukin, Commented Mar 6, 2018 at 3:40

Andrey Tyukin · Accepted Answer · 2018-03-06 03:44:36Z

1

The error message tells you that it cannot find an Encoder for a heterogeneous Array to save it in a Dataset. But you can get an RDD of Arrays like this:

cn_train.rdd.map{ row => row.toSeq.toArray }

answered Mar 6, 2018 at 3:44

Andrey Tyukin

45.1k4 gold badges65 silver badges103 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Scala - Encoder missing for type stored in dataset

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related