pyspark convert array to string in loop

Question

I have a psypark data frame which has string ,int and array type columns. I am trying to run a for loop for all columns to check if their is any array type column and convert it to string.

The output in the pyspark data frame should then hold the int,string columns.

The below code will return only the columns which were converted from array to string. How do i include else statement to get the remaining columns from dataframe which are not array type.

dfstring = df.select([(F.col(c).cast('String')).alias(c) for c in df.columns if dict(df.dtypes)[c] == 'array<string>'])

mck · Accepted Answer · 2021-04-10 20:06:43Z

1

You can try to modify the list comprehension as below:

dfstring = df.select([
    (F.col(c).cast('String')).alias(c)
    if 'array' in dict(df.dtypes)[c]
    else F.col(c)
    for c in df.columns 
])

answered Apr 10, 2021 at 20:06

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

pyspark convert array to string in loop

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related