How to concat a StringType column with every element of an ArrayType column in pyspark

Question

I have a StringType() column and an ArrayType(StringType()) column in a pyspark dataframe. I want to concat the StringType() column with every element of the ArrayType(StringType()) column.
Example:

+-----+---------------------+------------------------------+
|col1 |col2                 |col3                          |
+-----+---------------------+------------------------------+
|'AQQ'|['ABC', 'DEF']       |['AQQABC', 'AQQDEF']          |
|'APP'|['ABC', 'DEF', 'GHI']|['APPABC', 'APPDEF', 'APPGHI']|
+-----+---------------------+------------------------------+

Thanks :)

jxc · Accepted Answer · 2019-12-16 00:49:47Z

1

for spark 2.4+, use transform:

from pyspark.sql.functions import expr 

df = spark.createDataFrame([('AQQ', ['ABC', 'DEF']),('APP', ['ABC', 'DEF', 'GHI'])], ['col1', 'col2'])

df.withColumn('col3', expr("transform(col2, x -> concat(col1, x))")).show(truncate=False)                            
+----+---------------+------------------------+
|col1|col2           |col3                    |
+----+---------------+------------------------+
|AQQ |[ABC, DEF]     |[AQQABC, AQQDEF]        |
|APP |[ABC, DEF, GHI]|[APPABC, APPDEF, APPGHI]|
+----+---------------+------------------------+

answered Dec 16, 2019 at 0:49

jxc

14k4 gold badges20 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to concat a StringType column with every element of an ArrayType column in pyspark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related