I am trying to check if a dataframe is empty in Pyspark using below.
print(df.head(1).isEmpty)
But, I am getting an error
Attribute error: 'list' object has no attribute 'isEmpty'.
I checked if my object is really a dd using type(df) and it is class 'pyspark.sql.dataframe.Dataframe'
I used df.first() == None to evaluate if my spark dataframe is empty
df.first() is None would be the preferred syntax.When u do a head(1) it returns a list.
So that’s the reason for your error.
You have to just do df.isEmpty().
df.head(1) returns a list corresponding to the first row of df.
You can check if this list is empty "[ ]" using a bool type condition as in:
if df.head(1):
print("there is something")
else:
print("df is empty")
>>> 'df is empty'
Empty lists are implicity "False".
For better explanation, please head over to python docs.
Another way to do this would be to check if df.count()==0
Df.isEmpty would work for everyone starting from spark version 3.3.0. Earlier spark version might not support isEmpty method for pyspark.
df.rdd.isEmpty() instead. Source