Spark: Removing first array from Nested Array in Scala

Question

I have a DataFrame with 2 columns.I want to remove first array of the nested array in every record. Example :- I have a DF like this

+---+-------+--------+-----------+-------------+
|id |arrayField                                |
+---+------------------------------------------+
|1  |[[Akash,Kunal],[Sonu,Monu],[Ravi,Kishan]] |
|2  |[[Kunal, Mrinal],[Priya,Diya]]            |
|3  |[[Adi,Sadi]]                              |
+---+-------+---------+----------+-------------+

and I want my output like this:-

+---+-------+------+------+-------+
|id |arrayField                   |
+---+-----------------------------+
|1  |[[Sonu,Monu],[Ravi,Kishan]]  |
|2  |[[Priya,Diya]]               |
|3  | null                        |
+---+-------+------+------+-------+

notNull · Accepted Answer · 2020-08-17 18:39:51Z

2

From Spark-2.4 use slice function.

Example:

df.show(10,false)
/*
+------------------------+
|arrayField              |
+------------------------+
|[[A, k], [s, m], [R, k]]|
|[[k, M], [c, z]]        |
|[[A, b]]                |
+------------------------+
*/

import org.apache.spark.sql.functions._

df.withColumn("sliced",expr("slice(arrayField,2,size(arrayField))")).
withColumn("arrayField",when(size(col("sliced"))==0,lit(null)).otherwise(col("sliced"))).
drop("sliced").
show()
/*
+----------------+
|      arrayField|
+----------------+
|[[s, m], [R, k]]|
|        [[c, z]]|
|            null|
+----------------+
*/

answered Aug 17, 2020 at 18:39

notNull

31.8k4 gold badges41 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark: Removing first array from Nested Array in Scala

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related