How to explode array column given a formula?

Question

I am trying a very basic thing in spark scala, but couldn't figure it out.

I have data like this:

col1	col2
John	1
Jack	2

And, I want to achieve this:

That is, for each row, I want to create two more rows, one is val(col2)-1 and val(col2)+1.

I tried to use explode, but couldn't figure out how to do it properly.

val exploded_df = df.withColumn("col2", explode($"col2" -1, $"col2", $"col2" +1 ))

And, got:

too many arguments for method explode: (e: org.apache.spark.sql.Column)org.apache.spark.sql.Column

mck · Accepted Answer · 2021-01-22 15:43:01Z

1

You need to specify an array column to explode:

val exploded_df = df.withColumn("col2", explode(array($"col2" -1, $"col2", $"col2" +1 )))

answered Jan 22, 2021 at 15:43

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

1 Answer 1