python how to use subprocess pipe with linux shell

Question

I have a python script search for logs, it continuously output the logs found and I want to use linux pipe to filter the desired output. example like that:

$python logsearch.py | grep timeout

The problem is the sort and wc are blocked until the logsearch.py finishes, while the logsearch.py will continuous output the result.

sample logsearch.py:

p = subprocess.Popen("ping google.com", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
for line in p.stdout:
    print(line)

UPDATE:

figured out, just change the stdout in subprocess to sys.stdout, python will handle the pipe for you.

p = subprocess.Popen("ping -c 5 google.com", shell=True, stdout=**sys.stdout**)

Thanks for all of you help!

ping -c 5 is an odd example of a script that runs forever! But, yeah, you can't sort things or count the number of things until you have them all. — tdelaney
– tdelaney, Commented Oct 9, 2016 at 21:01
@tdelaney I removed the count for ping and instead of sort, I use grep so suppose the data should flow to the grep through pipe but it seems runs forever without print anything. — Vincent Mao
– Vincent Mao, Commented Oct 9, 2016 at 21:22
You don't receive nothing because the output don't match with timeout. Try $python logsearch.py | grep -v timeout — Jose Raul Barreras
– Jose Raul Barreras, Commented Oct 9, 2016 at 21:32

Community · Accepted Answer · 2017-05-23 12:16:32Z

1

And why use grep? Why don't do all the stuff in Python?

from subprocess import Popen, PIPE
p = Popen(['ping', 'google.com'], shell=False, stdin=PIPE, stdout=PIPE)

for line in p.stdout:
    if 'timeout' in line.split():
        # Process the error
        print("Timeout error!!")
    else:
        print(line)

UPDATE:
I change the Popen line as recommended @triplee. Pros and cons in Actual meaning of 'shell=True' in subprocess

edited May 23, 2017 at 12:16

CommunityBot

11 silver badge

answered Oct 9, 2016 at 21:37

Jose Raul Barreras

8601 gold badge13 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Vincent Mao Over a year ago

the reason need grep or other shell cmd is to add the flexibility, some time we might want add grep or sort or awk in order to better process the data.

Jose Raul Barreras Over a year ago

OK. Just in case, you can use the Python module sh. Implements a lot bash useful commands.

tripleee Over a year ago

Also use Popen(['ping', 'google.com'], shell=False, ...) to avoid spawning a useless shell.

Collectives™ on Stack Overflow

python how to use subprocess pipe with linux shell

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related