0

Spark documentation says that for scheduler allocation file we can: put a file named fairscheduler.xml on the classpath

I have included fairscheduler.xml in the root of the jar file that I use with spark-submit but pool scheduler is not used. Does anyone where to place fairscheduler.xml to be on the classpath? Thx

Improve this question
6
  • spark.scheduler.mode=FAIR . have you set this configuration? Commented Nov 1 at 5:26
  • I did set spark.scheduler.mode=FAIR and works fine locally with fairscheduler.xml in the resource directory, but when I upload the jar to Databricks it does not work. I am wondering if it's a Databricks bug. Thx Commented Nov 1 at 14:05
  • Okay. For databricks you might have to add in the spark config env variables under the jobs configuration Commented Nov 2 at 2:57
  • I did set spark.scheduler.mode=FAIR for Databricks as well, but fairscheduler.xml is not getting used from the .jar file and trying to reference it using workspace config("spark.scheduler.allocation.file","file:/Workspace/init/fairscheduler.xml") , it throws a token error. :( Commented Nov 2 at 12:55
  • I think asking in Databricks community will be helpful in that case. community.databricks.com. Commented Nov 3 at 4:34
Related questions
225
Add JAR files to a Spark job - spark-submit
188
How to turn off INFO logging in Spark?
183
How are stages split into tasks in Spark?
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.