Extract Array[T] from Spark dataframe in Scala

Question

I am trying to find a particular array (of type Double) based on another column with minimum value. The following code works to extract the array but I am unable to receive it as Array[Double]. Tried mapping and casting from as found from other threads but could not solve the problem. I would be grateful for any hints. The following is the illustration:

scala> df.show
+----+---------------+
|time|           crds|
+----+---------------+
|12.0|[0.1, 2.1, 1.2]|
| 8.0|[1.1, 2.1, 3.2]|
| 9.0|[1.1, 1.1, 2.2]|
+----+---------------+


scala> val minTime = df.select(min(col("time"))).collect()(0)(0).toString.toDouble
minTime: Double = 8.0

scala> val crd = df.filter($"time" === minTime).select($"crds").take(1)
crd: Array[org.apache.spark.sql.Row] = Array([WrappedArray(1.1, 2.1, 3.2)])

scala> val res: Array[Double] = crd.array
<console>:29: error: type mismatch;
 found   : Array[org.apache.spark.sql.Row]
 required: Array[Double]
   val res: Array[Double] = crd.array
                                ^

scala>

Ged · Accepted Answer · 2020-04-08 18:48:49Z

1

...
import scala.collection.mutable.WrappedArray
val crd = df.filter...select($"v").first.getAs[WrappedArray[Double]](0).toArray

answered Apr 8, 2020 at 18:48

Ged

18.5k8 gold badges53 silver badges108 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Quiescent · Accepted Answer · 2020-04-08 18:35:38Z

1

May be a but cumbersome but works assuming there is only one hit for minimum.

scala> val df = Seq(
     |    (12.0, Array(0.1, 2.1, 1.2)),
     |    (8.0, Array(1.1, 2.1, 3.2)),
     |    (9.0, Array(1.1, 1.1, 2.2))
     | ).toDF("time", "crds")
df: org.apache.spark.sql.DataFrame = [time: double, crds: array<double>]

scala> val minTime = df.select(min(col("time"))).collect()(0)(0).toString.toDouble
minTime: Double = 8.0

scala> val crd = df.filter($"time" === minTime).select(explode(col("crds"))).collect().map(i => i(0)).map(_.toString.toDouble)
crd: Array[Double] = Array(1.1, 2.1, 3.2)

scala>

answered Apr 8, 2020 at 18:35

Quiescent

1,19610 silver badges25 bronze badges

Collectives™ on Stack Overflow

Extract Array[T] from Spark dataframe in Scala

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related