Getting an issue when starting Spark on Kubernetes in client mode using JupyterHub.
21/10/05 03:54:33 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-0,5,main]
java.lang.NoSuchMethodError: 'java.lang.String org.slf4j.helpers.Util.safeGetSystemProperty(java.lang.String)'
at org.slf4j.impl.VersionUtil.getJavaMajorVersion(VersionUtil.java:11)
at org.slf4j.impl.Log4jMDCAdapter.<clinit>(Log4jMDCAdapter.java:37)
at org.slf4j.impl.StaticMDCBinder.getMDCA(StaticMDCBinder.java:59)
at org.slf4j.MDC.<clinit>(MDC.java:74)
at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$setMDCForTask(Executor.scala:740)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:432)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
21/10/05 03:54:33 INFO MemoryStore: MemoryStore cleared
21/10/05 03:54:33 INFO BlockManager: BlockManager stopped
21/10/05 03:54:33 INFO ShutdownHookManager: Shutdown hook called
I've also confirmed that the slf4j versions and jars are the same in the driver and executor as some articles suggest that it could be a package version mismatch.
Jars installed are
jcl-over-slf4j-1.7.30.jar
jul-to-slf4j-1.7.30.jar
slf4j-api-1.7.30.jar
slf4j-log4j12-1.7.30.jar
in both executor and driver pods. Would you know if there's something I missed? Thank you.
