DataFrame column (Array type) contains Null values and empty array (len =0). How to convert Null to empty array?

Use array function.

df = spark.createDataFrame([
  [None],   
  [[]],   
  [['foo']] 
]).toDF("a")

import pyspark.sql.functions as F

df.withColumn('a', F.coalesce(F.col('a'), F.array(F.lit(None)))).show(10, False)
+-----+
|a    |
+-----+
|[]   |
|[]   |
|[foo]|
+-----+

The result is now array(string), so there is no null value. Please check the results.

temp = spark.sql("SELECT a FROM table WHERE a is NULL")
temp.show(10, False)
temp = spark.sql("SELECT a FROM table WHERE a = array(NULL)")
temp.show(10, False)
temp = spark.sql("SELECT a FROM table")
temp.show(10, False)


+---+
|a  |
+---+
+---+

+---+
|a  |
+---+
|[] |
+---+

+-----+
|a    |
+-----+
|[]   |
|[]   |
|[foo]|
+-----+