Async execution of commands in Python

Question

Requirement - I want to execute a command that uses ls, grep, head etc using pipes (|). I am searching for some pattern and extracting some info which is part of the query my http server supports.

The final output should not be too big so m assuming stdout should be good to use (I read about deadlock issues somewhere) Currently, I use popen from subprocess module but I have my doubts over it.

how many simultaneous popen calls can be fired.
does the result immediately come in stdout? (for now it looks the case but how to ensure it if the commands take long time)
how to ensure that everything is async - keeping close to single thread model?

I am new to Python and links to videos/articles are also appreciated. Any other way than popen is also fine.

This is somewhat of an anti-pattern. If you just want to run a pileline, why not use a shell-script? Why do you even need Python? What exactly are you trying to accomplish? — Roland Smith
– Roland Smith, Commented Nov 3, 2015 at 15:55
The number of simultaneous "popen calls" will depend on OS limits and hardware resources. — dsh
– dsh, Commented Nov 3, 2015 at 16:13
@RolandSmith it is part of query processing of a http server. — singhsumit
– singhsumit, Commented Nov 3, 2015 at 16:23
one of the reason I posted the question is that I wanted to be as close single thread philosophy.. if needed thread-pools are fine but at the lowest level.. but may be there might be libraries that are not blocking and i could use them.. — singhsumit
– singhsumit, Commented Nov 3, 2015 at 16:28
What is your specific issue? An async. framework such as gevent, twisted, tornado, asyncio provide means to read subprocess' output asynchronously. Do you need a task queue such as celery? — jfs
– jfs, Commented Nov 4, 2015 at 14:34

Roland Smith · Accepted Answer · 2015-11-03 16:10:08Z

1

You could use os.listdir or os.walk instead of ls, and the re module instead of grep.

Wrap everything up in a function, and use e.g. the map method from a multiprocessing.Pool object to run several of those in parallel. This is a pattern that works very well.

In Python3 you can also use Executors from concurrent.futures in a similar way.

answered Nov 3, 2015 at 16:10

Roland Smith

43.8k3 gold badges69 silver badges98 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

singhsumit Over a year ago

can IOloop be used and executors be avoided?

Roland Smith Over a year ago

You mean from Tornado? I guess so, if you'd use Popen objects directly.

singhsumit Over a year ago

yes m using tornado. i meant schedule callbacks on ioLoop instead of executor(s).. will take be efficient? I couldnt get Popen related point..

Roland Smith Over a year ago

ioLoop is tied to file descriptors, isn't it? Not sure it would even work with a future.

Collectives™ on Stack Overflow

Async execution of commands in Python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related