Dynamically concat the dataframe columns in scala

Question

I have list of dataframe column names which I need to concat (list_name=["name","email"]) and I have a dataframe with many columns(df="name","email","address","phone"). Now I need to concat and create a new column for the values of names specified in list.

Expected result: df="name","email","address","phone","nameemail"

List=["name","email"]. But the list is dynamic(it may have n number of values)

df

name	email	phone
ram	[email protected]	345897045
raj	[email protected]	658086657

expexteddf

name	email	phone	nameemail
ram	[email protected]	345897045	[email protected]
raj	[email protected]	658086657	[email protected]

partlov · Accepted Answer · 2023-01-05 10:18:24Z

1

This should be straightforward using concat function. You should at least try and show what you did, but this should be way to go:

from pyspark.sql.functions import concat
val concatColumns = ... // List of column names to concatenate
val newColumnName = concatColumns.mkString
expexteddf = df.addColumn(newColumnName, concat(concatColumns: _*))

answered Jan 5, 2023 at 10:18

partlov

14.4k6 gold badges67 silver badges87 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Dynamically concat the dataframe columns in scala

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related