Python Subprocess / Environment Variable Script [duplicate]

Question

I have a compiled Fortran program (lets say in ~/codedir) linked against Intel MKL, which I want to run with a Python script in a certain directory (lets say calculationdir). I can navigate to calculationdir and execute ~/codedir/code, where code is the name of the executable. Everything works. But if I try to start the code from the below Python script:

import subprocess

command = '~/codedir/code'

with open('job.out', 'a') as f:
    process = subprocess.Popen(command.split(), stdout=f, shell=True)
    output, error = process.communicate()

I get the following error:

/home/codedir/code: error while loading shared libraries: libmkl_intel_lp64.so.1: cannot open shared object file: No such file or directory

I suspect that this might have something to do with the environment variables. I configured my shell that every time I start a shell, the script setvars.sh of Intel One API is executed which sets a lot of things. Could it be case that these variable are not set if I use python's subprocess? Do I have to tell subprocess in some way to execute also the setvars-script?

Do I have to tell subprocess in some way to execute also the setvars-script?. Most likely, yes you do have to. I'm a little surprised you didn't test that already. What shell are you using? — High Performance Mark
– High Performance Mark, Commented Apr 1, 2022 at 10:41
Thanks for your comment. My question would then be how to do that? The documentation of the subprocess module is a bit confusing to me. I am using tilix. — Hendrik
– Hendrik, Commented Apr 1, 2022 at 10:44
Did you export those variables? If you didn't, subpocesses will not see them. Set them before starting your Python script. — tripleee
– tripleee, Commented Apr 2, 2022 at 13:11

tripleee · Accepted Answer · 2022-04-02 13:04:40Z

You seem to be looking for

import subprocess
import shlex

with open('job.out', 'a') as f:
    subprocess.run(
        shlex.split(command),
        cwd=calculationdir,
        stdout=f, check=True)

There is no need to specify shell=True and in fact, when you split the command yourself, it's an error to do so on Unix-like platforms. (The semantics are slightly different on Windows, so while it's still an error, the symptoms are negligible.) See also Actual meaning of shell=True in subprocess

Plain split is wrong because it doesn't know how to cope with backslash escapes and quoting; if your command doesn't currently contain either of those constructs, you can get away with a basic string split; but the standard library supplies a function which works correctly regardless, and I see no reason not to use shlex.split here.

As the documentation explains, you should prefer subprocess.run over plain Popen when you can, as the latter requires you to copy/paste several lines of boilerplate code, and even then is less robust unless you know exactly what you are doing, probably at the expense of adding even more boilerplate code for error handling etc.

Finally, the actual beef: the keyword argument cwd allows you to set the working directory of the subprocess.

Collectives™ on Stack Overflow

Python Subprocess / Environment Variable Script [duplicate]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related