Python Subprocess Command as List Not String

Question

I need to use the subprocess module in Python to create some new files by redirecting stdout. I don't want to use shell=True because of the security vulnerabilities.

I wrote some test commands to figure this out, and I found that this worked:

import subprocess as sp
filer = open("testFile.txt", 'w')
sp.call(["ls", "-lh"], stdout=filer)
filer.close()

However, when I passed the command as one long string instead of a list, it couldn't find the file. So when I wrote this:

import subprocess as sp
filer = open("testFile.txt", 'w')
sp.call("ls -lh", stdout=filer)
filer.close()

I received this error:

Traceback (most recent call last):
  File "./testSubprocess.py", line 16, in <module>
    sp.call(command2, stdout=filer)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 524, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1308, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Why does it matter if I pass the arguments as a string or as a list?

glglgl · Accepted Answer · 2014-05-08 11:34:51Z

6

It is because of the way how the call happens:

With shell=True, the call is performed via the shell, and the command is given to the shell as one string.

With shell=False, the call is performed directly, via execv() and related functions. These functions expet an array of arguments.

If you only pass one string, it is treated as an abbreviation for a call with only the executable's name without arguments. But there is (probably) no executable called ls -lh on your system.

To be exact, somewhere deep inside subprocess.py, the following happens:

        if isinstance(args, types.StringTypes):
            args = [args]
        else:
            args = list(args)

So every string passed is turned into a list with one element.

        if shell:
            args = ["/bin/sh", "-c"] + args

This one I didn't know: obviously, this allows for passing additional arguments to the called shell. Although it is documented this way, don't use it as it is subject to create too much confusion.

If shell=False, we have further down below

if env is None:
    os.execvp(executable, args)
else:
    os.execvpe(executable, args, env)

which just takes a list and uses it for the call.

edited May 8, 2014 at 11:34

answered May 7, 2014 at 14:41

glglgl

91.5k13 gold badges157 silver badges230 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

jfs Over a year ago

One nitpick: do not even hint at the possibility of a list argument and shell=True together -- it is a common mistake, it is even present in PyMOTW page for the subprocess module.

glglgl Over a year ago

@J.F.Sebastian You are right, but I was sheerly interested in what happens and examined it thus.

jfs Over a year ago

unfortunately, it is documented. There is an issue on Python bug tracker: Popen should raise ValueError if pass a string when shell=False or a list when shell=True. The lesson is learned and the new API such as asyncio uses two different names instead *subprocess_exec(*args,...) and *subprocess_shell(cmd,...)

Brian Cain · Accepted Answer · 2014-05-07 14:47:46Z

If you want your strings to be divided like they would at a shell, use shlex:

import subprocess as sp
import shlex
with open("testFile.txt", 'w') as filer:
    sp.call(shlex.split("ls -lh"), stdout=filer)

BTW, let me make the case for check_call while I'm here. Without it, you'd get empty output if you added an invalid argument, for example. You'd be left wondering why the output at filer is empty.

with open("testFile.txt", 'w') as filer:
    sp.check_call(shlex.split("ls -lh0"), stdout=filer)

With check_call you get an error that localizes the problem and prevents subsequent code from executing:

Traceback (most recent call last):
  File "go.py", line 6, in <module>
    sp.check_call(shlex.split("ls -lh0"), stdout=filer)
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ls', '-lh0']' returned non-zero exit status 2

luk32 · Accepted Answer · 2014-05-07 14:39:07Z

3

It is because the argument is the interpreted as the executable name. It is the same if you put "ls -lh" into shell.

luk32:~/projects/tests$ "ls -lh"
bash: ls -lh: command not found

There is an utility for this called shlex.split.

>>> import shlex, subprocess
>>> command_line = raw_input()
/bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'"
>>> args = shlex.split(command_line)
>>> print args
['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"]
>>> p = subprocess.Popen(args)

But I don't think you need it. Just assume that going with list is the proper way, and the tool is to support transition from the discouraged shell=True mode.

answered May 7, 2014 at 14:39

luk32

16.1k2 gold badges42 silver badges67 bronze badges

Comments

perrocontodo · Accepted Answer · 2014-05-07 15:15:16Z

According to the comments in subprocess.py:

On UNIX, with shell=False (default): In this case, the Popen class uses os.execvp() to execute the child program. args should normally be a sequence. A string will be treated as a sequence with the string as the only item (the program to execute).

On UNIX, with shell=True: If args is a string, it specifies the command string to execute through the shell. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional shell arguments.

On Windows: the Popen class uses CreateProcess() to execute the child program, which operates on strings. If args is a sequence, it will be converted to a string using the list2cmdline method. Please note that not all MS Windows applications interpret the command line the same way: The list2cmdline is designed for applications using the same rules as the MS C runtime.

In UNIX subprocess.call('ls -l') will fail while in Windows it will succeed. It is os.execvp() which is causing the issue. The problem is that the whole string is passed as an argument. If you execute subprocess.call('free') it will succeed in UNIX.

Collectives™ on Stack Overflow

Python Subprocess Command as List Not String

4 Answers 4

3 Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related