How to convert ArrayType to DenseVector in PySpark DataFrame?

Question

I'm getting the following error trying to build a ML Pipeline:

pyspark.sql.utils.IllegalArgumentException: 'requirement failed: Column features must be of type org.apache.spark.ml.linalg.VectorUDT@3bfc3ba7 but was actually ArrayType(DoubleType,true).'

My features column contains an array of floating point values. It sounds like I need to convert those to some type of vector (it's not sparse, so a DenseVector?). Is there a way to do this directly on the DataFrame or do I need to convert to an RDD?

I was stuck on the same thing while calculating the vector norm. I used array_to_vector from pyspark.ml.functions to convert the array column to a vector type. Only available in pyspark>=3.1.0. More details here: stackoverflow.com/a/48333361/2650427. — TrigonaMinima
– TrigonaMinima, Commented Jun 1, 2023 at 14:08

zero323 · Accepted Answer · 2016-09-01 19:47:36Z

28

You can use UDF:

udf(lambda vs: Vectors.dense(vs), VectorUDT())

In Spark < 2.0 import:

from pyspark.mllib.linalg import Vectors, VectorUDT

In Spark 2.0+ import:

from pyspark.ml.linalg import Vectors, VectorUDT

Please note that these classes are not compatible despite identical implementation.

It is also possible to extract individual features and assemble with VectorAssembler. Assuming input column is called features:

from pyspark.ml.feature import VectorAssembler

n = ... # Size of features

assembler = VectorAssembler(
    inputCols=["features[{0}]".format(i) for i in range(n)], 
    outputCol="features_vector")

assembler.transform(df.select(
    "*", *(df["features"].getItem(i) for i in range(n))
))

edited Sep 1, 2016 at 19:47

answered Aug 18, 2016 at 19:38

zero323

331k108 gold badges982 silver badges958 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to convert ArrayType to DenseVector in PySpark DataFrame?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related