Is it possible to create nested virtual environments for python?

Question

While working on some automation scripts, I found the need to have some common packages available to all the scripts (like ipython and spyder) and installing only the, let's call it differential packages for each script. Is there a way to have nested virtual environment in python? That is, some way to set the environments so that python would first looks at the innermost environment; if not found, it would look on the second innermost environment and so on until reaching the system wide environment for a package?

The closest I could find was to use the venv module with the --system-site-packages option but I couldn't achieve what I looking for: first, I don't want to install packages on the system using root permissions; second, I couldn't find a way to nest virtual environments using this option.

What's the best way to achieve this nested virtual environments structure, if there's one?

palotasb · Accepted Answer · 2023-02-23 13:41:37Z

11

You can add a .pth file (a site module feature) to the site packages directory of your derived virtual environment with a line pointing to the site-packages path of your base virtual environment.

In shell, you can do it like this:

# Assumes that the base virtual environment exists, activate it.
. base/bin/activate

# Create the derived virtual environment.
python -m venv ./derived

# Make the derived virtual environment import base's packages too.
base_site_packages="$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')"
derived_site_packages="$(./derived/bin/python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')"
echo "$base_site_packages" > "$derived_site_packages"/_base_packages.pth

^{base_site_packages is usually base/lib/python<VERSION>/site-packages, the code to get it is taken from https://stackoverflow.com/a/46071447/3063 – same for derived_site_packages.}

The packages installed in the base environment will be available in the derived environment. You can verify this by doing pip list in the derived environment.

# Deactivating the base environment is optional,
# meaning that the derived environment can be activated directly too.
deactivate

. ./derived/bin/activate
pip list

answered Feb 23, 2023 at 13:41

palotasb

4,7183 gold badges31 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Anas Rzq Over a year ago

main ├── method_1_folder │ ├── method_1_github_repository_cloned_folder │ └── venv_1 ├── method_2_folder │ ├── method_2_github_repository_cloned_folder │ └── venv_2 └── running_python_file.py └── main_venv during execution of running_python_file.py in main_venv, I am trying to switch to venv_1 within python script using following code

#directory at method_1_folder level   subprocess.run("deactivate", shell=True) -> return /bin/sh: 1: deactivate: not found subprocess.call(f"source {os.path.join('venv', 'bin', 'activate')}", shell=True) -> return /bin/sh: 1: source: not found

wim · Accepted Answer · 2020-04-04 03:14:41Z

5

No, there isn't any such "nesting" feature in venv.

I think your best bet would be to define the common packages in a requirements file and then install those at the time when you create a new virtual environment

pip install -r requirements-common.txt

answered Apr 4, 2020 at 3:14

wim

368k114 gold badges681 silver badges816 bronze badges

6 Comments

user13079354 Over a year ago

That's the approach I'm currently using. My complaint about this method is that the total size of my virtual environment is around 3Gb for 23 scripts of less than 1K each. Anyway, if there's no way, that's the answer.

wim Over a year ago

If you don't mind a bit of a hack, you could put all the common libs into a directory on disk somewhere and then set the PYTHONPATH environment variable. That should be visible in all venvs.

sinoroc Over a year ago

I would use a .pth file rather than modifying PYTHONPATH. See this answer and its links for some ideas on how to achieve that: stackoverflow.com/a/60974302/11138259 -- in particular this: discuss.python.org/t/…

wim Over a year ago

@sinoroc Unless I'm missing something obvious, a path file would have to be put in each venv's site. That seems more of a hassle than setting the env var once. Why rather a .pth file?

wim Over a year ago

I see. The point of .pth files is to allow an installed package to dynamically extend it's own import locations at install time. In OP's case would seem like a bit of an abuse of that mechanism, so I wouldn't recommend to use path files here.

|

Ildar · Accepted Answer · 2022-07-12 03:53:46Z

1

It is possible with nix-shell. I use it just like this - nix-shell <first shell.nix>. Then if needed - in the same session nix-shell <the second shell.nix>

Development environment with nix-shell

So, second session uses underlying first session

edited Jul 12, 2022 at 3:53

answered Jul 12, 2022 at 3:46

Ildar

192 bronze badges

Collectives™ on Stack Overflow

Is it possible to create nested virtual environments for python?

3 Answers 3

1 Comment

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related