Pyspark - struct to string to multiple columns

Question

I have a dataframe with a schema as follows:

root
 |-- column: struct (nullable = true)
 |    |-- column-string: string (nullable = true)
 |-- count: long (nullable = true)

What I want to do is:

Get rid of the struct - or by that I mean "promote" column-string, so my dataframe only has 2 columns - column-string and count
I then want to split column-string into 3 different columns, so I end up with the schema:

The text within column-string always fits the format: Some-Text,Text,MoreText

Does anyone know how this is possible?

I'm using Pyspark Python.

PS. I am new to Pyspark & I don't know much about the struct format and couldn't find how to write an example into my post to make it reproducible - sorry.

mck · Accepted Answer · 2021-02-16 17:03:29Z

1

You can also use from_csv to convert the comma-delimited string into a struct, and then star expand the struct:

import pyspark.sql.functions as F

df2 = df.withColumn(
    'col',
    F.from_csv(
        'column.column-string',
        '`column-string` string, `column-string2` string, `column-string3` string'
    )
).select('col.*', 'count')

df2.show()
+-------------+--------------+--------------+-----+
|column-string|column-string2|column-string3|count|
+-------------+--------------+--------------+-----+
|     SomeText|          Text|      MoreText|    1|
+-------------+--------------+--------------+-----+

Note that it's better not to have hyphens in column names because they are reserved for subtraction. Underscores are better.

answered Feb 16, 2021 at 17:03

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

blackbishop · Accepted Answer · 2021-02-16 16:43:26Z

0

You can select column-string field from the struct using column.column-string, the simply split by a comma to get three columns :

from pyspark.sql import functions as F

df1 = df.withColumn(
    "column_string", F.split(F.col("column.column-string"), ",")
).select(
    F.col("column_string")[0].alias("column-string"),
    F.col("column_string")[1].alias("column-string2"),
    F.col("column_string")[2].alias("column-string3"),
    F.col("count")
)

edited Feb 16, 2021 at 16:43

answered Feb 16, 2021 at 16:38

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Collectives™ on Stack Overflow

Pyspark - struct to string to multiple columns

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related