Running a shell script using Subprocess in Python does not produce output

Question

I am trying to run a shell script using through Python using subprocess.Popen().

The shell script just has the following lines:

#!/bin/sh
echo Hello World

Following is the Python code:

print("RUNNNING SHELL SCRIPT NOW")
shellscript = subprocess.Popen(['km/example/example1/source/test.sh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
shellscript.wait()

for line in shellscript.stdout.readlines():
    print(line)
print("SHELL SCRIPT RUN ENDED")

However, on running this, I am only getting the following output:

RUNNNING SHELL SCRIPT NOW
SHELL SCRIPT RUN ENDED

i.e. I am not getting the shell script output in between these 2 lines.

Moreover, when I remove the stderr=subprocess.PIPE part from the subprocess, I get the following output:

RUNNNING SHELL SCRIPT NOW
'km' is not defined as an internal or external command.
SHELL SCRIPT RUN ENDED

I am not able to understand how to resolve this, and run the shell script properly. Kindly guide. Thanks.

UPDATE:

I also tried the following change:

print("RUNNNING SHELL SCRIPT NOW")
shellscript = subprocess.Popen(['km/example/example1/source/test.sh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)

out, err = shellscript.communicate()
print(out)
print("SHELL SCRIPT RUN ENDED")

I get the following output:

RUNNNING SHELL SCRIPT NOW
b''
SHELL SCRIPT RUN ENDED

Usually the first to do when using subprocess, is to check if shellscript.returncode == 0 otherwise there as was an error (details should be available in err). The command passed to Popen has to be a string when using shell=True, whereas shell=False requires the command as list — Maurice Meyer
– Maurice Meyer, Commented Nov 23, 2021 at 7:23
You are using a relative path (not starting with /). Nothing wrong with that provided you make sure what the current working directory is. When using an IDE it is often not what you would expect. (Just add import os and print(os.getcwd) to control it...) — Serge Ballesta
– Serge Ballesta, Commented Nov 23, 2021 at 7:31
use shell=True and pass the command as string not list and one more thing your path is not correct provide absolute path to the script from very begining e.g. /var/www/file.html — Zain Ul Abidin
– Zain Ul Abidin, Commented Nov 23, 2021 at 7:31
@ZainUlAbidin That is precisely incorrect advice. You want to avoid shell=True when you can. — tripleee
– tripleee, Commented Nov 23, 2021 at 9:27

tripleee · Accepted Answer · 2021-11-23 09:31:37Z

2

The simple and straightforward fix is to not use bare Popen for this.

You also don't need a shell to run a subprocess; if the subprocess is a shell script, that subprocess itself will be a shell, but you don't need the help of the shell to run that script.

proc = subprocess.run(
    ['km/example/example1/source/test.sh'],
    check=True, capture_output=True, text=True)
out = proc.stdout

If you really need to use Popen, you need to understand its processing model. But if you are just trying to get the job done, the simple answer is don't use Popen.

The error message actually looks like you are on Windows, and it tries to run km via cmd which thinks the slashes are option separators, not directory separators. Removing the shell=True avoids this complication, and just starts a process with the requested name. (This of course still requires that the file exists in the relative file name you are specifying. Perhaps see also What exactly is current working directory? and also perhaps switch to native Windows backslashes, with an r'...' string to prevent Python from trying to interpret the backslashes.)

edited Nov 23, 2021 at 9:31

answered Nov 23, 2021 at 9:26

tripleee

192k37 gold badges318 silver badges367 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

eraisedtothepowerx Over a year ago

Thank you for the answer. Yes I don't really need to use Popen. Yes, I am on Windows. When I run your code snippet, I get ` OSError: [WinError 193] %1 is not a valid Win32 application ` What could this be related to?

tripleee Over a year ago

If km/example/example1/source/test.sh is not a valid Windows application, that would be the explanation. I don't know how you would expect shell scripts to work on Cygwin; maybe explicitly run it with ['/bin/sh', 'km/example/example1/source/test.sh']?

eraisedtothepowerx Over a year ago

I had to provide the complete path starting from the Drive. However, when the script runs, a terminal window open for just a second displays the output and closes itself. Is there a way to display the output on the Pycharm terminal instead, so that it stays there?

tripleee Over a year ago

No experience with PyCharm and very little of Python on Windows; but this sounds like a common FAQ. Ask a new question if you can't find it already answered.

Collectives™ on Stack Overflow

Running a shell script using Subprocess in Python does not produce output

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related