PySpark - Fill in null values in a Struct column

Question

I have the following dataframe:

+---+---------+
| ID|    Title|
+---+---------+
|  1|[2, test]|
|  3|     [4,]|
+---+---------+

created using the code below

from pyspark.sql.types import StructType,StructField, StringType, IntegerType
from pyspark.sql.functions import col, expr
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

data = [(1, [2, 'test']), (3, [4, None])]

schema = (StructType([ 
    StructField("ID",IntegerType(),False),   
    StructField("Title",StructType([
      StructField("TitleID",IntegerType(),False),
      StructField("Type",StringType(),True),
    ]),False) 
  ]))

df = spark.createDataFrame(data, schema)

Now I'm trying to replace the null title types with a default value. I have tried this using fillna but it doesn't have any effect:

default_type = 'type one'
df = df.fillna({'Title.Type':default_type})

I have also tried using a expr

df = df.withColumn('Title', expr('struct(Title.TitleID, Title.Type if Title.Type.isNotNull() else default_type'))

but now I get a ParseException:

ParseException: 
extraneous input 'Title' expecting {')', ','}(line 1, pos 36)

== SQL ==
struct(Title.TitleID, Title.Type if Title.Type.isNotNull() else default_type
------------------------------------^^^

What am I doing wrong here?

Connell.O'Donnell · Accepted Answer · 2021-05-19 10:56:12Z

1

You're confusing Spark SQL expr with Python expr:

import pyspark.sql.functions as F

df = df.withColumn(
    'Title', 
    F.expr(f"struct(Title.TitleID as TitleID, case when Title.Type is not null then Title.Type else '{default_type}' end as Type)")
)

edited May 19, 2021 at 10:56

Connell.O'Donnell

3,74312 gold badges36 silver badges67 bronze badges

answered May 19, 2021 at 10:50

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

PySpark - Fill in null values in a Struct column

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related