48

Let's say I have 2 different versions of my app installed in 2 different virtualenvironments. myapp v1.0 and myapp v2.0.

Now I would like to compare those. The comparison is written in python itself. What would be the best way to do that? Let's assume I can run them separately and both write an output file, which I can compare later.

One way to do that would be to write a bash script (that's what I have currently). I activate one virtualenv, run myapp v1.0, activate another virtualenv, run myapp v2.0. Later run a comparison module on those files. But I would like to add some more dynamics there (take some optional arguments etc) which would be easier with python.

Currently I have something like that (a bash script):

source virtualenv1/bin/activate
python my_script.py
deactivate

source virtualenv2/bin/activate
python my_other_script.py
deactivate

python my_comparison_script.py

instead, I'd like to do only:

python my_comparison_script.py

and my scripts would be run inside this.

Improve this question

4 Answers 4

Reset to default
75

If you start your application with a call to the python executable, like in your example it is actually very simple: you only have to explicitly point to the executable in the virtualenv.

import subprocess

subprocess.Popen(["virtualenv1/bin/python", "my_script.py"])
subprocess.Popen(["virtualenv2/bin/python", "my_other_script.py"])

will start the processes in the respective virtualenvs.

Important

To address the concerns voiced in the comments:

If you want to run a subprocess and be sure to use the same interpreter that the current process is running in you have to use sys.executable. Also available: sys.exec_prefix to access the site-specific directory prefix where the platform-dependent Python files are installed.

If you want a much more in depth discussion of this topic, have a look at this pull request.

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

Actually, this is not enough. Activating a virtualenv also changes some environment variables, calling specific virutalenv's python doesn't do that. As a result, wrong libraries will be loaded.
@qarma: I disagree - this approach works and I am using it in my own production code. I am not the only one recommending it, either. I would be helpful if you concretize the environment variables you are referring to and provide a test case that shows under wich circumstances wrong libraries would be loaded.
Yes I've seen that recommendation on virtualenv's site too. There are at least two cases where running with virtualenv linked python interpreter directly is different from using activate: when PYTHONHOME is set and when target program uses command or Popen to run another Python program. The latter if arguably murky.
I just tried the second scenario and that's true: calling subprocess from inside my_script.py would be exectuted with the system Python again. Awkward.
On windows. I need to "set PYTHONPATH= & workon virtualenv1" before calling python or the script. Otherwise, the python interpreter PYTHONPATH still points back to the system libraries.
|
17

I think virtualenv documentation explains it nicely.

TL;DR

Runnig python venv binary directly is NOT the same as activating venv. You also have to change PATH and VIRTUAL_ENV variables accordingly (look at os.environ)

Source

$ source /path/to/ENV/bin/activate

This will change your $PATH so its first entry is the virtualenv’s bin/ directory. (You have to use source because it changes your shell environment in-place.) This is all it does; it’s purely a convenience.

If you directly run a script or the python interpreter from the virtualenv’s bin/ directory (e.g. path/to/ENV/bin/pip or /path/to/ENV/bin/python-script.py) then sys.path will automatically be set to use the Python libraries associated with the virtualenv. But, unlike the activation scripts, the environment variables PATH and VIRTUAL_ENV will not be modified. This means that if your Python script uses e.g. subprocess to run another Python script (e.g. via a !/usr/bin/env python shebang line) the second script may not be executed with the same Python binary as the first nor have the same libraries available to it. To avoid this happening your first script will need to modify the environment variables in the same manner as the activation scripts, before the second script is executed.

Improve this answer

Comments

12

A simple option would be to run a series of commands with subprocess as follows (note that shell=True is risky and should only be used if you can control the input).

import subprocess
    
cmd = 'source activate my_virtualenv; python my_script.py'
subprocess.call(cmd, shell=True, executable='/bin/bash')

And repeat as needed.

Improve this answer

3 Comments

Source won't be available inside subprocess because subprocess uses /bin/sh. This will give the error: "/bin/sh: 1: source: not found"
Use . instead of source, which is the portable equivalent
Perhaps highight that the effects of source will disappear when subprocess.call finishes. The main meat here is running your main Python script in the same subprocess as the source command, which is often unattractive for other reasons. (Maybe instead then run a shell script which does the source and then runs your entire Python script.)
0

Oh dear. None of these answers is right.

There is a huge difference between running a python executable in your VE and activating that VE.

To do the latter in Linux:

os.system(". Projects/virenv/bin/activate && python Projects/virenv/django-project/manage.py runserver")

To give credit where it's due this is from this answer.

In W10 you may be able to use os.system (I haven't tried) but you can also use subprocess.run with the same tell-tale "&&", which allows multiple commands in one line:

If start_ve.bat merely contains your path to your "activate" file (in W10 this runs another .bat file called activate.bat): "D:\apps\Python\virtual_envs\dev_doc_indexer\Scripts\activate"

Then:

subprocess.run('start_ve.bat && python run_my_thing.py', shell=True)

It doesn't seem possible to do this in Linux (i.e. using a .sh file): the first command must (I believe) always start with the sourcing ".".

I'm not sure whether "shell=True" is necessary in the above version.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.