How to add an empty array using when and otherwise in pyspark

Question

How can i add an empty array when using df.withColomn when() and otherwise(***empty_array***)
New column type is T.ArrayType(T.StringType()) from UDF

I want to avoid ending up with NaN values.

Already tried that but get an error: Unsupported literal type class java.util.Arraylist [] — John Doe
– John Doe, Commented Aug 4, 2020 at 10:26

Shubham Jain · Accepted Answer · 2020-08-04 10:35:58Z

2

Simply use array(lit(None))

df.select(when(col('target_bool')=='true',array(lit(1))).otherwise(array(lit(None)))).show()

answered Aug 4, 2020 at 10:35

Shubham Jain

5,6162 gold badges20 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

John Doe Over a year ago

Is there a difference between F.array([]) and F.array(F.lit(none)) ?

Shubham Jain Over a year ago

Using lit for literals is recommended. Both are correct otherwise

dsk · Accepted Answer · 2020-08-04 10:41:04Z

0

Try below - Create a column with None value and cast to Array()

df_b = df_b.withColumn("empty_array", F.when(F.col("rn") == F.lit("1"), (None))).withColumn("empty_array", F.col("empty_array").cast(T.ArrayType(T.StringType())))
df_b.show()



 root
 |-- col1: string (nullable = true)
 |-- col2: string (nullable = true)
 |-- rn: integer (nullable = true)
 |-- case_condition: integer (nullable = true)
 |-- empty_array: array (nullable = true)
 |    |-- element: string (containsNull = true)

edited Aug 4, 2020 at 10:41

answered Aug 4, 2020 at 10:33

dsk

2,0272 gold badges12 silver badges13 bronze badges

Collectives™ on Stack Overflow

How to add an empty array using when and otherwise in pyspark

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related