How to convert a DataFrame with String into a DataFrame with Vectors in Scala(Spark 2.0)

Question

I have a DataFrame with a column named KFA containing a string with angular braces on both ends. There are 4 double values in this long string. I would like to convert this into a DataFrame with vectors.

This is the first element of the DataFrame:

> dataFrame1.first()
res130: org.apache.spark.sql.Row = [[.00663 .00197 .29809 .0034]]

Could you help me to covert it into a dense vector with 4 double values.

I have tried this command

dataFrame1.select("KFA")
    .map((x=>x.mkString("").replace("]","").replace("[","").split(" ")))
    .rdd.map(x=>Vectors.dense(x(0).toDouble,x(1).toDouble,x(2).toDouble,x(3).toDouble,x(4).toDouble))

This looks very clumsy and unreadable. Could you suggest any other ways of doing this?

What's the purpose of the mkString if you are just going to split it? — OneCricketeer
– OneCricketeer, Commented Dec 5, 2016 at 21:37
@cricket_007 Used mkString because I couldn't use .replace("]","") on a spark.sql.Row — Anbarasu
– Anbarasu, Commented Dec 6, 2016 at 13:37
You don't need to create a string, I don't think. You can getAs[Double] from a Row object — OneCricketeer
– OneCricketeer, Commented Dec 6, 2016 at 14:59

akuiper · Accepted Answer · 2016-12-05 23:10:41Z

1

Here is an option with Regular expression:

import org.apache.spark.mllib.linalg.{Vector, Vectors}

val p = "[.0-9]+".r 
val rddVec = dataFrame1.select("KFA")
             .map(x => Vectors.dense(p.findAllIn(x(0).toString).map(_.toDouble).toArray))

# rddVec: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector] = MapPartitionsRDD[49] at map at <console>:39

rddVec.collect
# res43: Array[org.apache.spark.mllib.linalg.Vector] = 
         Array([0.00663,0.00197,0.29809,0.0034], [0.00663,0.00197,0.29809,0.0034])

answered Dec 5, 2016 at 23:10

akuiper

216k33 gold badges362 silver badges379 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to convert a DataFrame with String into a DataFrame with Vectors in Scala(Spark 2.0)

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related