Scala Spark DataFrame Map Encoder non-primitive type

Question

I have a Scala Spark DataFrame (Variable df):

id, values
"a", [0.5, 0.6]
"b", [0.1, 0.2]
...

I am trying to make use of RowMatrix to calculate pairwise cosine similarity efficiently.

final case class dataRow(id: String, values: Array[Double])

val rows = df.as[dataRow].map {
  row => {
        Vectors.dense(row.values)
    }
}.rdd

I am having the following compilation error

Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._

Eventually, I would be able to do this (RowMatrix requires an RDD[Vector])

val mat = new RowMatrix(rows)

I have already imported spark.implicits_, what am I doing wrong?

Does the id matter in any way? Note that RowMatrix will not keep the order between the rows, if that is important use IndexedRowMatrix instead. — Shaido
– Shaido, Commented Feb 12, 2019 at 9:04
@Shaido You're right, I do need the id so that I can identify which are the pairs. — Ivan
– Ivan, Commented Feb 12, 2019 at 10:05

user11050468 · Accepted Answer · 2019-02-12 11:26:53Z

1

There is simply no implicit Encoder for Vector types. So either push map after `rdd

val rows = df.as[dataRow].rdd.map(row => Vectors.dense(row.values))

or provide an Encoder

import org.apache.spark.sql.Encoder
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder

ds.as[dataRow].map(x => Vectors.dense(x.values))(ExpressionEncoder(): Encoder[Vector])

answered Feb 12, 2019 at 11:26

user11050468

92 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Joyoyoyoyoyo · Accepted Answer · 2019-02-12 09:18:08Z

0

Which Vectors object are you using?

Try importing the linalg context. There might be conflicts within the libraries.

Also move the case class domain object outside of your function scope and then remove final

import org.apache.spark.mllib.linalg.{Vectors, Vector}
import org.apache.spark.mllib.linalg.distributed.RowMatrix

case class DataRow(id: String, values: Array[Double])


def func(spark: SparkSession, df: DataFrame): RowMatrix = {
   import spark.implicits._

   val rows = df.as[DataRow]
      .map(row => Vectors.dense(row.values))
      .rdd

   val mat = new RowMatrix(rows)
   mat
}

edited Feb 12, 2019 at 9:18

answered Feb 12, 2019 at 9:10

Joyoyoyoyoyo

1651 gold badge3 silver badges19 bronze badges

5 Comments

Ivan Over a year ago

I'm using import org.apache.spark.ml.linalg.{Vector, Vectors}. Case class is outside the scope, I am still getting the same compilation error

Joyoyoyoyoyo Over a year ago

Which version of spark are you using? Odd, because your code very much resmbles that of the cosine similarity: github.com/apache/spark/blob/master/examples/src/main/scala/org/…

Ivan Over a year ago

thats exactly what I'm trying to do with this library, except trying to overcome the encoding error.

Joyoyoyoyoyo Over a year ago

@Ivan where do you insert 'import spark.implicits._'? can you show us in your example

Joyoyoyoyoyo Over a year ago

@Ivan I think you can try selecting df and cast "id" as a string, and values as an Array[Double]

Collectives™ on Stack Overflow

Scala Spark DataFrame Map Encoder non-primitive type

2 Answers 2

Comments

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related