pyspark concat_ws for array to string

Question

I have a pyspark data frame which has string ,int and array type columns. I am trying to run a for loop for all columns to check if their is any array type column and convert it to string.

The output in the pyspark data frame should then hold the int,string columns.

I have below code however it fails with below error. Any suggestions to fix this error

df2 = df.select([df.withColumn(c,F.concat_ws(",",col(c))).alias(c)  if dict(df.dtypes)[c] in 'array<string>' else F.col(c) for c in df.columns])

Error : For column literals, use 'lit', 'array', 'struct' or 'create_map' function.

mck · Accepted Answer · 2021-04-11 15:08:48Z

4

You cannot put withColumn inside select. Try the code below:

import pyspark.sql.functions as F

df2 = df.select([
    F.concat_ws(",", c).alias(c)  
    if dict(df.dtypes)[c] == 'array<string>'
    else F.col(c)  
    for c in df.columns
])

answered Apr 11, 2021 at 15:08

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

pyspark concat_ws for array to string

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related