Running a GNU parallel command using python subprocess

Question

Here is a simple GNU parallel command that creates a file called "example_i.txt" inside an existing directory called "example_i". It does this four times, for i from 1 to 4, with one job per core:

parallel -j 4 'cd example_{} && touch example_{}.txt' ::: {1..4}

Not very exciting, I know. The problem appears when I try to run this via python (v3.9) using the subprocess module as follows:

import subprocess
cmd = "parallel -j 4 'cd example_{} && touch example_{}.txt' ::: {1..4}"
subprocess.run(cmd, shell=True)

When doing so I get this error:

/bin/sh: 1: cd: can't cd to example_{1..4}

It looks like using the python subprocess call, bash is not triggering the call correctly as a GNU parallel command. Instead, it is substituting the {1..4} explicitly rather than dividing it up into four jobs.

I also tried this with the less advisable os.system(cmd) syntax and got back the same error.

PS: For context, this question stems from me trying to use UQpy (the RunModel module in particular) for uncertainty quantification of a Fortran code that was handed to me. Although this is not directly related to the question, it is relevant because I would like to know how to get this working using these tools as I am not at liberty to change them.

I think you either need to ensure subprocess.run() uses bash, or avoid "bashisms" by replacing {1..4} with 1 2 3 4. — Mark Setchell
– Mark Setchell, Commented Sep 15, 2021 at 22:12

Marc · Accepted Answer · 2021-09-16 08:20:19Z

3

Following @Mark Setchell's comment, indeed it appears that bash is not used by default on POSIX as can be seen in the documentation for subprocess. This is solved by explicitly telling subprocess to use bash by re-writting my python code snippet as:

import subprocess
cmd = "parallel -j 4 'cd example_{} && touch example_{}.txt' ::: {1..4}"
subprocess.run(cmd, shell=True, executable='/bin/bash')

It should be noted that although the argument executable is here being used in the subprocess.run() call, it is not directly a part of this class. The executable argument is actually part of the subprocess.Popen() class, but it is accessible to subprocess.run() by the **other_popen_kwargs argument.

answered Sep 16, 2021 at 8:20

Marc

515 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Running a GNU parallel command using python subprocess

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related