0

I keep running into this issue when running PySpark.

I was able to connect to my database and retrieve data, but whenever I try do operations like .show() or .count(), or when I try to save a Spark DataFrame to a CSV, it keeps crashing with the following error traceback.

(Note: I am using SparkSession.builder)

Error 1:

py4j.protocol.Py4JJavaError: An error occurred while calling o121.count.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 13 in stage 0.0 failed 1 times, most recent failure: Lost task 13.0 in stage 0.0 (TID) ( driver): org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)
        at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:624)
        at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:599)

Error 2:

25/09/29 15:21:22 WARN TaskSetManager: Lost task 1.0 in stage 2.0 (TID) ( executor driver): TaskKilled (Stage cancelled: Job aborted due to stage failure: Task 4 in stage 2.0 failed 1 times, most recent failure: Lost task 4.0 in stage 2.0 (TID 4) (executor driver): org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)
        at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:624)
        at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:599)
Improve this question
3
  • 1
    You can probably find more details about the error in stdout log produced by python runner itself than in stderr printed by Spark driver. Commented Sep 29 at 14:39
  • Per mazaneicha, please look in the driver logs to see what actually happened and use this to create a question that people can answer. As it stands there is no real information for people here to be able to help. Commented Sep 30 at 10:22
  • the key detail in the error is Python worker exited unexpectedly (crashed). this could be due to multiple reasons - one of which is incompatible python or java version; another is incorrect configs. try looking into those. Commented Sep 30 at 10:33
Related questions
0
Error from PySpark code to showdataFrame : py4j.protocol.Py4JJavaError
0
fail DF show pyspark
18
Apache Spark: pyspark crash for large dataset
Load 4 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.