Error: When convert spark dataframe to pandas dataframe

Question

I need to convert spark dataframe (large datasets) into pandas dataframe.

Code : spark_df = Example_df.toPandas()

I am getting this error:

/databricks/spark/python/pyspark/sql/pandas/conversion.py:145: UserWarning: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true, but has reached the error below and can not continue. Note that 'spark.sql.execution.arrow.pyspark.fallback.enabled' does not have an effect on failures in the middle of computation.

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 30 tasks (31.0 GiB) is bigger than local result size limit 30.0 GiB, to address it, set spark.driver.maxResultSize bigger than your dataset result size.

pltc · Accepted Answer · 2022-03-02 21:36:50Z

1

TL;DR: The error is clear, you need to set spark.driver.maxResultSize so something bigger than 31Gb.

Longer answer: when running toPandas, you're requesting Spark to demand all executors to send data back to a single driver, and the driver's memory has to be big enough to hold that much data.

answered Mar 2, 2022 at 21:36

pltc

6,0371 gold badge16 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Error: When convert spark dataframe to pandas dataframe

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related