Troubleshooting errors

This page explains gives a list of common errors and how to troubleshoot and fix them.

My Spark application never finishes

You need to call spark.stop() at the end of your application code, where spark can be your Spark session or Spark context. Otherwise your application may keep running indefinitely.

Python worker exited unexpectedly (crashed)

If you see this message in the Spark UI or in the Driver logs, it means that a Python process running inside a Spark executor has crashed while running a Spark task. The full error typically looks like this:

21/05/18 00:22:38 WARN TaskSetManager: Lost task 1.0 in stage 2.0 (TID 10) (172.16.3.13 executor 1): org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)

...
Caused by: java.io.EOFException

The root cause of this issue is that the container was running out of memory, and so the operating system decided to interrupt a Python process to free up some memory.

To fix it, yous should increase your container memory overhead. Increase your container memory size, for example by choosing a memory-optimized instance type, can also help.