Exploding Array of Tuple in a dataframe column in Spark Scala

Question

I want to explode Array[(Int, Int)] column from a dataframe

INPUT:

colA newCol
1     [[1a, 2],[3c, 5u]]
2     [[1c, 9m], [5e, 7l]]

OUTPUT:

colA newCol
1     1a
1     3c
2     1c
2     5e

Please use text instead of images. You should also add what have you tried and the error, if you encountered any. For more information, read How to Ask. — philantrovert
– philantrovert, Commented Aug 1, 2019 at 6:40

Daeho Ro · Accepted Answer · 2019-08-01 05:07:15Z

2

Here is my approach.

+----+--------------------+
|col1|col2                |
+----+--------------------+
|1   |[[1a, 2b], [3c, 5u]]|
|2   |[[1c, 9m], [5e, 7l]]|
+----+--------------------+

This is your dataframe and

df.withColumn("t", explode($"col2")).selectExpr("col1", "t[0]").show

my code results to

+----+----+
|col1|t[0]|
+----+----+
|   1|  1a|
|   1|  3c|
|   2|  1c|
|   2|  5e|
+----+----+

answered Aug 1, 2019 at 5:07

Daeho Ro

13.7k4 gold badges25 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Raja Sabarish PV Over a year ago

How can we solve it if t[0] column is of IntegerType

Daeho Ro Over a year ago

Let me give your desired dataframe.

Raja Sabarish PV Over a year ago

Was trying something like this: val res = df.withColumn("tup", explode($"newCol")).select("colA", "tup") res.select(col("colA"), col("tup")("value").as("uId"))

Collectives™ on Stack Overflow

Exploding Array of Tuple in a dataframe column in Spark Scala

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related