How can one python script launch another python script using a different virtual environment?

Question

I have researched this question and found Running subprocess within different virtualenv with python, but my situation is a little different.

I have an astrophotography application which I run on Ubuntu Linux 24.04.2 LTS. This application has a facility for launching python scripts. It does so using its own venv. They supply a python script to launch a third party executable that sharpens an image that the main application is working with. This third party executable optionally can use GPU acceleration given the presence of a suitable GPU on the system running the script. This GPU detection functionality depends on code in the main application's venv.

Another possibly important detail is that the third-party executable is itself compiled from a python script, which presumably uses a venv. It is not c++ or other similar language.

The situation is when the script is run from the application, an error message is emitted saying that GPU acceleration is not available, and the process drops back to CPU processing which takes perhaps perhaps 100 times as long. This error message is incorrect.

I know this because the third party executable can be successfully run standalone when the much faster GPU processing can be observed.

To do that one must manually copy the file to be processed to a specific directory where the standalone executable expects to find it. If running from the application, the existing python script automatically copies the selected file to the executable's input directory, while also copying the processed file back from the executable's output directory to the same directory where the original image was located, a very convenient feature.

The problem lies with the application's python venv, which incorrectly detects whether or not the GPU support is available. Apparently, this is one big thorny mess which the application's developers are working on. When run standalone, the executable uses the system's python venv (remember it is compiled python).

In the meantime I would like to develop a script that can be run from the application that handles all the file copying before and after launching another python script using the system venv which launches the executable. In other words, to avoid the problematic GPU detection logic, since I know that in my case the GPU is suitable.

Is this possible and if so, what would be the best way to do it?

Lluciocc · Accepted Answer · 2025-09-12 18:17:53Z

3

Yes it’s possible. The general approach is that your script (running inside the application’s venv) doesn’t try to import or execute the other Python directly, but instead "spawns" another interpreter from the correct virtual environment. That way you sidestep the GPU-detection mess in the application’s venv.

here some code that can help you:

import subprocess
import shutil
import os

def main(input_file, output_file):
    gpu_python = "/home/you/gpuenv/bin/python"
    gpu_script  = "/path/to/the/script_or_entrypoint.py"

    # copy input file to where the GPU tool expects it
    workdir = "/tmp/gpu_work"
    os.makedirs(workdir, exist_ok=True)
    local_input = os.path.join(workdir, os.path.basename(input_file))
    shutil.copy2(input_file, local_input)

    # call the GPU enabled script in its own .venv
    result = subprocess.run(
        [gpu_python, gpu_script, local_input],
        capture_output=True, text=True
    )
    print(result.stdout)
    print(result.stderr)

    # copy the result back
    produced_file = os.path.join(workdir, "output.fits")  # adjust as needed
    shutil.copy2(produced_file, output_file)

answered Sep 12 at 18:17

Lluciocc

312 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Steve Cohen Sep 12 at 20:29

Thanks, @Lluciocc! But here is a question. Does local_input need to be passed to the subprocess that will invoked by running gpu_script? The executable that will be launched by gpu_script already knows where to look for its input.

Lluciocc Sep 24 at 14:11

Hi, It depends on how the gpu_script is written. If it expects the input file as a command-line argument, then yes, you need to pass local_input to it. But if the script already knows where to look for its input (e.g. it always reads from /tmp/gpu_work), then you don’t need to provide the path, just copy the file there and run the script without arguments.

Collectives™ on Stack Overflow

How can one python script launch another python script using a different virtual environment?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related