How to run shell-exectued python script in a virtual environment

Question

I have a shell script that executes some python code, and installs the necessary dependencies to do so beforehand:

sudo -E PATH="$PATH" python3 -m pip install -r requirements.txt

The script runs successfully when executed like so:

./script.sh

but I get an import error from Python when I run the script with sudo permission:

sudo ./script.sh

Traceback (most recent call last):
  File "dsc.py", line 2, in <module>
    from imblearn.over_sampling import ADASYN
  File "/usr/local/lib/python3.8/dist-packages/imblearn/__init__.py", line 53, in <module>
    from . import ensemble
  File "/usr/local/lib/python3.8/dist-packages/imblearn/ensemble/__init__.py", line 8, in <module>
    from ._forest import BalancedRandomForestClassifier
  File "/usr/local/lib/python3.8/dist-packages/imblearn/ensemble/_forest.py", line 28, in <module>
    from sklearn.utils.fixes import _joblib_parallel_args
ImportError: cannot import name '_joblib_parallel_args' from 'sklearn.utils.fixes' (/usr/local/lib/python3.8/dist-packages/sklearn/utils/fixes.py)

I don't understand why Python can no longer import ADASYN when the script is given sudo permission.

I am using virtualenv for version control. When I check where the imblearn package is located when in the same virtual environment from where the shell script was exectued, I get:

❯ python3 -m pip show imblearn
WARNING: Package(s) not found: imblearn

But I can find it when not in a virtual environment:

❯ deactivate
❯ python3 -m pip show imblearn
Name: imblearn
Version: 0.0
Summary: Toolbox for imbalanced dataset in machine learning.
Home-page: https://pypi.python.org/pypi/imbalanced-learn/
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Location: /usr/local/lib/python3.8/dist-packages
Requires: imbalanced-learn
Required-by:

Which means that my shell script does install the dependencies, but not within the virtual environment from which it is executed. Removing the imblearn package and running the shell script installs it again, so I am certain that it is this shell script that installs the package to the default location. I thought that by running Python code with the sudo -E PATH="$PATH" option makes sure that the python command has the same path as the virtual environment from which it is run, and thus the correct dependencies. But given that the dependencies get installed in my default Python package directory /usr/local/lib/python3.8/dist-packages, this seems not to be the case.

So how can I run python code in a shell script within the same virtual environment from which the shell script was executed? Is this possible at all? I am familiar with Python, but very new to shell.

I'm using Python 3.8.10 (both as default and in the virtualenv), Ubuntu 20.04 and zsh.

hide1nbush · Accepted Answer · 2022-08-31 07:08:50Z

1

If you run sudo ./script.sh, the actually executor of the script is the superuser, which means none of the env-related stuff in your virtual env is synced over.

You may need to specify the python virtual env inside your ./script.sh, i.e., adding one line on the top of the ./script.sh: #!path/to/your/virtualenvs/python

answered Aug 31, 2022 at 7:08

hide1nbush

9451 gold badge4 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Bjorge Over a year ago

Thanks for your reply. I found that virtualenv can be synced over to sudo using the -E PATH=$PATH option (see other answer). Adding #!path/to/your/virtualenvs/python at the top does not help sadly; I'm getting a syntax error for the function defined the line below:

./run_automated_DS_project.sh test12 1000   File "./run_automated_DS_project.sh", line 5     echo "Add description of the script functions here."          ^ SyntaxError: invalid syntax

Bjorge · Accepted Answer · 2022-08-31 09:41:07Z

1

The problem can be mitigated by running the bash script like so:

sudo -E PATH=$PATH ./script.sh

instead of just sudo ./script.sh.

This way, sudo executes the shell script, and the python path of your current virtualenv is passed along.

Inside the shell script, you now don't need to execute all python scripts with sudo -E PATH=$PATH script.py, but simply using python script.py suffices, since in this case python refers to the virtualenv python version.

edited Aug 31, 2022 at 9:41

answered Aug 31, 2022 at 7:41

Bjorge

1167 bronze badges

Collectives™ on Stack Overflow

How to run shell-exectued python script in a virtual environment

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related