How to asynchronously call a shell script from Python?

Question

I have a shell script which does some processing over the string passed and then writes it to a file. However I don't want my function foo() to wait for it to complete the operation. How do I call process(msg) and then move on the with the execution of {code block 2} without waiting for process(msg) to complete execution?

def process(msg):
    subprocess.call(['sh', './process.sh', msg])

def foo():
    # {code block 1}
    process(msg)
    # {code block 2}

foo() will be called from another function, almost once or twice per second.

CampFireMan · Accepted Answer · 2021-10-25 19:21:09Z

3

Just for completeness: Python's asyncio offers a high level interface for doing just that: https://docs.python.org/3.9/library/asyncio-subprocess.html#subprocesses

Example from documentation:

import asyncio

async def run(cmd):
    proc = await asyncio.create_subprocess_shell(
        cmd,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE)

    stdout, stderr = await proc.communicate()

    print(f'[{cmd!r} exited with {proc.returncode}]')
    if stdout:
        print(f'[stdout]\n{stdout.decode()}')
    if stderr:
        print(f'[stderr]\n{stderr.decode()}')

asyncio.run(run('ls /zzz'))

answered Oct 25, 2021 at 19:21

CampFireMan

762 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Paul Becotte · Accepted Answer · 2021-09-26 21:45:04Z

1

subprocess.call() and subprocess.run() creates a process, waits for it to finish, and returns a CompletedProcess object.

subprocess.Popen() creates a process and returns it. It is used under the hood of the previous functions. You can then wait for the process to finish, send it messages, or whatever else you want to do with it. The arguments are mostly the same as to call or run.

https://docs.python.org/3/library/subprocess.html

As a bit of elaboration, Popen is the python implementation of using the os to start a new process. os.fork() is a lower level that doesn't actually do what we want here, that would spawn another instance of the python interpreter with the same memory state as the current one. If you wanted to use the lower level syscall, os.spawn is closer to subprocess.run than os.fork.

To verify that Popen is doing what you want, this test program will pring "returncode = None", then wait 5 seconds, and print "returncode = 0"

from subprocess import Popen

p = Popen(["sleep", "5"])
print("started the proc")  # this will print immediately
p.poll()  # this checks if the process is done but does not block
print(f"p returncode = {p.returncode}")
p.wait()  # this blocks until the process exits
print(f"p returncode = {p.returncode}")

edited Sep 26, 2021 at 21:45

answered Sep 26, 2021 at 17:20

Paul Becotte

9,9973 gold badges36 silver badges43 bronze badges

3 Comments

malfoytool Over a year ago

so if I understand it right, if I use subprocess.popen(['sh', './process.sh', msg]) the python program would continue execution without waiting for the ./process.sh' to complete?

shiv Over a year ago

No. The process.sh will block the parent process. If you want async then you need something like os.fork()

Paul Becotte Over a year ago

That is correct. popen will not block the parent process until you make one of the blocking api calls such as wait or communicate. subprocess.run() and subprocess.call() do block the parent process.

shiv · Accepted Answer · 2021-09-26 18:27:41Z

0

What you need is https://docs.python.org/3/library/os.html#os.fork i.e. os.fork() that way you can spawn a child which can outlive the parent process which can be later claimed by systemd on Linux. I have no clue about Windows.

answered Sep 26, 2021 at 18:27

shiv

1,9721 gold badge17 silver badges24 bronze badges

Collectives™ on Stack Overflow

How to asynchronously call a shell script from Python?

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related