Spark Dataframe - Computation of pairs between columns (Scala)

Question

I have the following situation: I have a dataframe with an 'id' and 'array' as the schema. Now I want to get for each array, all lists of pairs with the corresponding id and save it again in a dataframe. So for example:

This is the original dataframe:

+---+----------+
| id|candidates|
+---+----------+
|  1|    [2, 3]|
|  2|       [3]|
+---+----------+

And that is how it have to look like after the computation:

+---+---+
|id1|id2|
+---+---+
|  1|  2|
|  1|  3|
|  2|  3|
+---+---+

Maybe someone has an idea for this problem?

Simply use the explode function

cheseaux
– cheseaux

2018-11-06 15:48:35 +00:00
Commented Nov 6, 2018 at 15:48 — cheseaux
– cheseaux, Commented Nov 6, 2018 at 15:48
And how I can use it for all array elements?

nuwuwa
– nuwuwa

2018-11-06 15:52:21 +00:00
Commented Nov 6, 2018 at 15:52 — nuwuwa
– nuwuwa, Commented Nov 6, 2018 at 15:52

nuwuwa · Accepted Answer · 2018-11-06 15:58:03Z

1

Ok, thanks @cheseaux I found the answer! There is the simply explode_outer function:

    candidatesDF.withColumn("candidates", explode_outer($"candidates")).show

answered Nov 6, 2018 at 15:58

nuwuwa

10310 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

cheseaux Over a year ago

Use explode if you don't want any row when the array is null/empty. if you use explode_outer it will in that case produce null, which might not be desired.

Belwal · Accepted Answer · 2020-05-12 18:13:49Z

0

Simply explode the array column.

candidatesDF.withColumn("id2", explode('candidates))

answered May 12, 2020 at 18:13

Belwal

4831 gold badge4 silver badges16 bronze badges

Collectives™ on Stack Overflow

Spark Dataframe - Computation of pairs between columns (Scala)

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related