Python subprocess - write multiple stdin

Question

I need to open an R script and supply it with input formulated by a separate python script. The subprocess module seems to be a good way to do this.

I have encountered some puzzling results though, namely that I can apparently write once and only once via p.stdin. Here is what I have so far:

from subprocess import Popen, PIPE, STDOUT

p = Popen(['r --no-save'],stdin=PIPE,stdout=PIPE,stderr=PIPE,shell=True)
p.stdin.write("source('myrscript.R')\n")
p.stdin.write('myfirstinput')

What happens when I run this code is that the first instance of stdin.write() performs as expected (and opens my R script), but the second line does nothing, and the subprocess (really, the R script) exits with an error, indicating that the subprocessed received no input where input was expected and therefore terminated.

N.B. - In a perfect world, I would just interact directly through R, but this particular script requires complex inputs that cannot be entered directly for practical purposes. Also, rpy / rpy2 is not an option, because end-users of this script will not necessarily have access to that module or its dependencies. rscript is also not an option (for many reasons, but mainly because of variability in the end-users R configurations).

Finally, p.communicate is not an option, because apparently that will close the process after writing and I need to keep it open.

Thanks in advance

good question, and no, including p.stdin.flush() between the write commands does does not change the result. — J R
– J R, Commented Apr 3, 2013 at 14:57
According to the subprocess module documentation for Popen... Note Do not use stdout=PIPE or stderr=PIPE with this function. As the pipes are not being read in the current process, the child process may block if it generates enough output to a pipe to fill up the OS pipe buffer. Maybe something wacky is going on in the script? Have you tried just issuing 2 simple commands like, p.stdin.write("a = 7") p.stdin.write("print(a)") instead of the script? — kith
– kith, Commented Apr 3, 2013 at 15:01
That was a helpful suggestion, thanks. It does indeed appear to be something internal to my script, because when I issue 2 separate commands, as you suggest, I get the expected behavior from p.stdin.write(). I will troubleshoot the R script - many thanks! — J R
– J R, Commented Apr 3, 2013 at 17:13
Uhm, ok. Do you have access to this script? At least the relevant parts where the standard input is read? To see why I asked check the notes at the end of help(stdin) — vodka
– vodka, Commented Apr 3, 2013 at 18:09

Hai Vu · Accepted Answer · 2014-04-11 22:26:20Z

6

What you need is to call .communicate():

from subprocess import Popen, PIPE, STDOUT

p = Popen(
    ['r', '--nosave'],
    stdin=PIPE,
    stdout=PIPE,
    stderr=PIPE)
p.stdin.write("source('myrscript.R')\n")
p.stdin.write('myfirstinput\n')
p.stdin.write('q\n')

stdout, stderr = p.communicate()

print '---STDOUT---'
print stdout
print '---STDERR---'
print stderr
print '---'

Discussion

I don't use the shell=True and it seems working with my fake R script since I don't have R install in my system. You might or might not need it.
I prefer breaking the command line up into a list of string as shown, but a single string such as r --nosave will work as well; just don't do them both at the same time.
Don't forget that stdin.write() does not write the new line character \n, you have to supply that yourself.

Update

My first attempt was off the mark, I hope this second attempt gets closer. As J.F. Sebastian suggested, you might want to use pexpect:

import pexpect
import sys

if __name__ == '__main__':
    prompt = '> ' # Don't know what the R prompt looks like
    lines = ['one', 'two', 'three']

    r = pexpect.spawn('r --no-save', logfile=sys.stdout)
    for line in lines:
        r.expect(prompt)
        r.sendline(line)

    # If you want to interact with your script, use these two lines
    # Otherwise, comment them out
    r.logfile = None # Turn off logging to sys.stdout
    r.interact()

Discussion

You might need to install pexpect. I did it with pip install pexpect
If you don't want to interact with the system, comment out the last two line, but make sure to send some signal for the R script to exit.
spawn() returns a spawn object, see doc here.

edited Apr 11, 2014 at 22:26

answered Apr 11, 2014 at 21:46

Hai Vu

41.4k16 gold badges75 silver badges106 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

jfs Over a year ago

OP already said that p.communicate() is not suitable because it waits for the process to complete. Also use p.communicate("\n".join(["source('myrscript.R')", "myfirstinput", "q"])) instead of p.stdin.write() calls otherwise you may cause a deadlock

Collectives™ on Stack Overflow

Python subprocess - write multiple stdin

1 Answer 1

Discussion

Update

Discussion

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Discussion

Update

Discussion

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related