Create empty column of StructType in spark dataframe

Question

I need to add an empty column of StructType to an existing DataFrame.

Tried following:

df = df.withColumn("features", typedLit(StructType(Nil)))

And:

df = df.withColumn("features", lit(new GenericRowWithSchema(Array(), StructType(Nil))))

However, in both of the above cases getting an error as unsupported literal type.

SaiNageswar S · Accepted Answer · 2018-11-06 14:04:14Z

1

In a crude way, one can use a user-defined function to add a column with empty rows:

def addEmptyRowColumn(df: DataFrame, newColumnName: String): DataFrame = {
  val addEmptyRowUdf = udf( () =>
    new GenericRowWithSchema(Array(), StructType(Nil)), StructType(Nil))

  df.withColumn(newColumnName, addEmptyRowUdf())
}

df = addEmptyRowColumn(df, "features")

answered Nov 6, 2018 at 14:04

SaiNageswar S

1,35615 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

NickyPatel Over a year ago

Though technically this answer is right, as per the spark developer community and spark tuning tips, it is not a good practice to use UDF. I would suggest to consider the above answer than this answer.

Prometheus · Accepted Answer · 2022-06-01 08:42:30Z

1

In a single liner and without UDF: from pyspark.sql import types as T, functions as F

df.withColumn(newColumnName, F.lit(None).cast(T.StructType()))

edited Jun 1, 2022 at 8:42

answered Mar 25, 2022 at 8:38

Prometheus

5647 silver badges19 bronze badges

2 Comments

insanely_sin Over a year ago

Whats F and T ?

Prometheus Over a year ago

thank you for the headsup. I updated the sample: from pyspark.sql import types as T, functions as F

Collectives™ on Stack Overflow

Create empty column of StructType in spark dataframe

2 Answers 2

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related