Submit Python script to Databricks JOB

Question

Is it possible to submit/configure a spark python script (.py) file to databricks job?

I have my developments happening in my Pycharm IDE, then push/commit the code to our gitlab repository. My requirement is I need to create new jobs in databricks cluster as and when a python script is moved to a GitLab master branch.

I would like to get some suggestions if its possible to create a databricks job on a python script, using gitlab.yml scripts?

In databricks Job UI, I could see spark jar or a notebook that can be used, but wondering if we can provide a python file.

Thanks,

Yuva

Raphael K · Accepted Answer · 2019-09-06 00:22:33Z

5

This functionality is not currently available in the Databricks UI, but it is accessible via the REST API. You'll want to use the SparkPythonTask data structure.

You'll find this example in the official documentation:

curl -n -H "Content-Type: application/json" -X POST -d @- https://<databricks-instance>/api/2.0/jobs/create <<JSON
{
  "name": "SparkPi Python job",
  "new_cluster": {
    "spark_version": "5.2.x-scala2.11",
    "node_type_id": "i3.xlarge",
    "num_workers": 2
  },
  "spark_python_task": {
    "python_file": "dbfs:/docs/pi.py",
    "parameters": [
      "10"
    ]
  }
}JSON

If you need help getting started with the REST API, see here.

answered Sep 6, 2019 at 0:22

Raphael K

2,3831 gold badge18 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Yuva Over a year ago

Thanks Raphael, can it be done from gitlab by any chance? Just curious

Collectives™ on Stack Overflow

Submit Python script to Databricks JOB

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related