Convert array<string> into string pyspark dataframe

Question

I have a pyspark dataframe where some of its columns contain array of string (and one column contains nested array). As a result, I cannot write the dataframe to a csv.

Here is an example of the dataframe that I am dealing with -

    +-------+--------------------+---------+
    |ID     |             emailed| clicked
    +-------+--------------------+---------+
    |9000316|[KBR, NRT, AOR]     |[[AOR]]  
    |9000854|[KBR, NRT, LAX]     | Null 
    |9001996|[KBR, JFK]          |[[JFK]] 
    +-------+--------------------+---------+

I would like to get the following structure, to be saved as a csv.

    +-------+--------------------+---------+
    |ID     |             emailed| clicked
    +-------+--------------------+---------+
    |9000316|KBR, NRT, AOR       | AOR  
    |9000854|KBR, NRT, LAX       | Null 
    |9001996|KBR, JFK            | JFK 
    +-------+--------------------+---------+

I am very new to pyspark. Your help is greatly appreciated. Thank you!

Will the column clicked always have this format - [[value]] or it can be [[val1,val2...]] ? — Spandan Brahmbhatt
– Spandan Brahmbhatt, Commented Sep 11, 2017 at 17:51

T.SURESH ARUNACHALAM · Accepted Answer · 2020-06-25 16:14:22Z

6

Can you try this way. You will have to import the module

import pyspark.sql.functions.*
df.select(concat_ws(',', split(df.emailed)).alias('string_form')).collect()

Let me know if that helps.

-----Update----

Code explained in the link, I modified a bit.

from pyspark.sql.functions import *
from pyspark.sql.types import *

def getter(column):
    col_new=''
    for i,col in enumerate(column):
        if i==0:
           col_new=col
        else:
           col_new=col_new+','+col
    return col_new

getterUDF = udf(getter, StringType())

df.select(getterUDF(Ur_Array_Column))

You can try this as well.

edited Jun 25, 2020 at 16:14

T.SURESH ARUNACHALAM

2852 silver badges15 bronze badges

answered Sep 11, 2017 at 18:24

Manu Gupta

8456 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Manu Gupta Over a year ago

Not sure if the above ans will work. As I checked, split works here on on string. You can view this as well on Stack Overflow. stackoverflow.com/questions/37689878/…

Manu Gupta Over a year ago

You can use this function:

T.SURESH ARUNACHALAM Over a year ago

You have to change i==0 instead of i==1

Manu Gupta Over a year ago

agreed. Editing the same. Thanks

Collectives™ on Stack Overflow

Convert array<string> into string pyspark dataframe

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related