Concat multiple columns of a dataframe using pyspark

Question

Suppose I have a list of columns, for example:

col_list = ['col1','col2']
df = spark.read.json(path_to_file)
print(df.columns)
# ['col1','col2','col3']

I need to create a new column by concatenating col1 and col2. I don't want to hard code the column names while concatenating but need to pick it from the list.

How can I do this?

cph_sto · Accepted Answer · 2019-02-28 19:17:00Z

27

You can use pyspark.sql.functions.concat() to concatenate as many columns as you specify in your list. Keep on passing them as arguments.

from pyspark.sql.functions import concat
# Creating an example DataFrame
values = [('A1',11,'A3','A4'),('B1',22,'B3','B4'),('C1',33,'C3','C4')]
df = sqlContext.createDataFrame(values,['col1','col2','col3','col4'])
df.show()
+----+----+----+----+
|col1|col2|col3|col4|
+----+----+----+----+
|  A1|  11|  A3|  A4|
|  B1|  22|  B3|  B4|
|  C1|  33|  C3|  C4|
+----+----+----+----+

In the concat() function, you pass all the columns you need to concatenate - like concat('col1','col2'). If you have a list, you can un-list it using *. So (*['col1','col2']) returns ('col1','col2')

col_list = ['col1','col2']
df = df.withColumn('concatenated_cols',concat(*col_list))
df.show()
+----+----+----+----+-----------------+
|col1|col2|col3|col4|concatenated_cols|
+----+----+----+----+-----------------+
|  A1|  11|  A3|  A4|             A111|
|  B1|  22|  B3|  B4|             B122|
|  C1|  33|  C3|  C4|             C133|
+----+----+----+----+-----------------+

edited Feb 28, 2019 at 19:17

answered Feb 28, 2019 at 9:09

cph_sto

7,70714 gold badges49 silver badges91 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

G.v. Sridhar Over a year ago

can we have a delimiter in between

badger Over a year ago

@G.v.Sridhar you can use something like this concat(col("a"), lit(" "), col("b")) and in lit() you can put your delimiter

Rich Dudley Over a year ago

If you want a delimiter, use concat_ws, like this (* is the delimiter): df = df.withColumn('concatenated_cols',concat_ws('*',*col_list))

Sudhir Jangam Over a year ago

when I use concat_ws(",", *col_list), it returns "_1" as col heading instead of "concatenated_cols" and the value becomes Row(concatenated_cols='A1,11,A3,A4) .. how do I fix this ?

Collectives™ on Stack Overflow

Concat multiple columns of a dataframe using pyspark

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related