Convert Array[DenseVector] to CSV with Scala

Question

I am using Kmeans Spark function with Scala and I need to save the Cluster Centers obtained into a CSV. This val is type: Array[DenseVector].

val clusters = KMeans.train(parsedData, numClusters, numIterations)
val centers = clusters.clusterCenters

I was trying converting centers to a RDD file and then from RDD to DF, but I get a lot of problems (e.g, import spark.implicits._ / SQLContext.implicits._ is not working and I cannot use .toDF). I was wondering if there is another way to make a CSV easier.

Any suggestion?

Denis Iakunchikov · Accepted Answer · 2018-01-14 18:53:38Z

4

Without use of external libraries you can do that by simply writing to the file Java way.

import java.io.{ PrintWriter, File, FileOutputStream }

...

val pw = new PrintWriter(
    new File( "KMeans_centers.csv" )
)

centers
.foreach( vec =>
        pw.write( vec.toString.drop( 1 ).dropRight( 1 ) + "\n" )
    )

pw.close()

Resulting file

0.1,0.1,0.1
9.1,9.1,9.1

drop and dropRight are needed to remove [] around the converted vector.

Code and data are taken from the official example.

answered Jan 14, 2018 at 18:53

Denis Iakunchikov

1,3492 gold badges11 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Convert Array[DenseVector] to CSV with Scala

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related