Installing Custom Python Wheel Packages in a Databricks Serverless Notebook

Question

I have some custom functions and classes that I packaged as a Python wheel. I want to use them in my python notebook (with a .py extension) that runs on a serverless Databricks cluster.

I have read that it is not recommended to use %pip install directly on serverless cluster. Instead, dependencies should be managed through the environment configuration panel, which is located on the right-hand side of the notebook interface. However, this environment panel works when the notebook file has a .ipynb extension, not when it is a .py file.

Given this, is it recommended to use %pip install inside a .py file running on a serverless platform, or is there a better way to manage custom dependencies like Python wheels in this scenario?

CHEEKATLAPRADEEP · Accepted Answer · 2025-08-27 14:05:25Z

0

You can manage Python dependencies for your notebook using the Environment side panel. There are two main ways to add them:

To add a dependency manually:

Open the Environment panel in your notebook.
Under Dependencies, click Add Dependency.
Enter the path to the dependency. You can use:
- A standard package name (like in a requirements.txt file),
- A .whl file,
- Or a Python project folder (with pyproject.toml or setup.py).
Path formats:
- For workspace files: use an absolute path starting with /Workspace/.
- For Unity Catalog volumes: use /Volumes/<catalog>/<schema>/<volume>/<path>.whl.
Click Apply to install the dependency and restart the Python environment.

For more details, refer to Configure the serverless environment

answered Aug 27 at 14:05

CHEEKATLAPRADEEP

12.8k1 gold badge22 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Alexis Kan Aug 29 at 14:13

Is this approach valid when running a job? I have hard time to define the databricks asset bundle using serverless with python wheel

Collectives™ on Stack Overflow

Installing Custom Python Wheel Packages in a Databricks Serverless Notebook

1 Answer 1

To add a dependency manually:

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

To add a dependency manually:

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related