How to merge multiple spark dataframe columns in one list useing Scala?

Question

I have the following Spark dataframe in Scala:

+---------+--------------------+--------------------+--------------------+
|       id|           col_str_1|           col_str_2|            col_list|
+---------+--------------------+--------------------+--------------------+
|        1|                   A|                   C|              [E, F]|
|        2|                   B|                   D|              [G, H]|
+---------+--------------------+--------------------+--------------------+

Where col_str_1 and col_str_2 are of type Stirng, and col_list is of type List[String].

I want a way to transform this dataframe into the following:

+---------+--------------------+
|       id|            col_list|
+---------+--------------------+
|        1|        [E, F, A, C]|
|        2|        [G, H, B, D]|
+---------+--------------------+

Any idea? Thank you.

mck · Accepted Answer · 2021-04-20 14:26:55Z

2

You can use concat to append elements to the array column:

val df2 = df.select(
    col("id"), 
    concat(
        col("col_list"), 
        array(col("col_str_1"), col("col_str_2"))
    ).as("col_list")
)

df2.show
+---+------------+
| id|    col_list|
+---+------------+
|  1|[E, F, A, C]|
|  2|[G, H, B, D]|
+---+------------+

answered Apr 20, 2021 at 14:26

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to merge multiple spark dataframe columns in one list useing Scala?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related