Py(Spark) udf gives PythonException: 'TypeError: 'float' object is not subscriptable

Question

I am applying user defined function to spark dataframe as below,

@udf("double")
def discount_udf (row):
  if ((row['total_order'] == 2) or (row['total_order'] == 3)):
    return 2.50
  elif ((row['total_order'] == 4) or (row['total_order'] == 5)):
    return 1.20
  elif ((row['total_order'] == 6) or (row['total_order'] == 7)):
    return 0.60
  elif ((row['total_order'] == 8) or (row['total_order'] == 9) or (row['total_order'] == 10) or (row['total_order'] == 11)):
    return 0.00
  elif ((row['total_order'] == 12) or (row['total_order'] == 13) or (row['total_order'] == 14) or (row['total_order'] == 15)):
    return -0.20
  elif ((row['total_order'] == 16) or (row['total_order'] == 17) or (row['total_order'] == 18) or (row['total_order'] == 19) or (row['total_order'] == 20) or (row['total_order'] == 21) or (row['total_order'] == 22) or (row['total_order'] == 23)):
    return -0.20
  elif ((row['total_order'] == 24) or (row['total_order'] == 25) or (row['total_order'] == 26) or (row['total_order'] == 27) or (row['total_order'] == 28) or (row['total_order'] == 29) or (row['total_order'] == 30) or (row['total_order'] == 31)):
    return -0.40
  else :
    return -0.50

from pyspark.sql.functions import udf
df.withColumn("discount_rate", discount_udf(F.col('total_order')))

However, this gives me this error

Error

PythonException: 'TypeError: 'float' object is not subscriptable', from <command-1374686736879751>, line 3. Full traceback below:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 100.0 failed 4 times, most recent failure: Lost task 0.3 in stage 100.0 (TID 164) (10.139.64.4 executor 0): org.apache.spark.api.python.PythonException: 'TypeError: 'float' object is not subscriptable', from <command-1374686736879751>, line 3. Full traceback below:
Traceback (most recent call last):
  File "<command-1374686736879751>", line 3, in discount_udf
TypeError: 'float' object is not subscriptable

I have tried same function with `pandas' dataframe it works fine for me.

df['discount_rate_1'] = df.apply(discount_udf, axis=1)

Can anybody help/suggest what is wrong here?

Thanks in advance

Steven · Accepted Answer · 2021-08-19 09:07:56Z

1

I change the begining, and let you do the rest :

@udf("double")
def discount_udf (total_order):
  if ((total_order == 2) or (total_order == 3)):

You just need to replace each row['total_order'] with total_order.

I'd advice you also to do this change :

if ((total_order == 2) or (total_order == 3)):
# TO BECOME
if total_order  in (2,3):
# OR ALSO POSSIBLE
if 2 <= total_order <= 3: # It is not exactly the same but should work if you only have integer

edited Aug 19, 2021 at 9:07

answered Aug 19, 2021 at 9:01

Steven

15.4k7 gold badges49 silver badges80 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Py(Spark) udf gives PythonException: 'TypeError: 'float' object is not subscriptable

Error

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Error

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related