val schema = StructType(Array(StructField("id", IntegerType, false),StructField("num", IntegerType, false)))
I want to generate continuous number from 0 to num by every id。 I don't know how to do .. Thanks
-
Added code formatanoop4real– anoop4real2017-04-13 07:58:17 +00:00Commented Apr 13, 2017 at 7:58
Add a comment
|
2 Answers
You can use UDF and explode function:
import org.apache.spark.sql.functions.{udf, explode}
val range = udf((i: Int) => (0 to i).toArray)
df.withColumn("num", explode(range($"num")))
Comments
Try DataFrame.explode:
df.explode(col("id"), col("num")) {case row: Row =>
val id = row(0).asInstanceOf[Int]
val num = row(1).asInstanceOf[Int]
(0 to num).map((id, _))
}
Or in RDD land, you can use flatmap for this:
df.rdd.flatMap(x => (0 to x._2).map((x._1, _)))
