Using python subprocess with module load

Question

I'm currently using Python 2.7 on a unix environment. I need to run R scripts in my python scripts but I can't manage to make it work because my R module needs to be loaded before (using "module load")

Here's my python script :

import os
import subprocess as sp

os.system('module load R/3.2.3')
out = sp.check_output(['Rscript','test.R'], universal_newlines=True)

I keep having the same error : "[Errno 2] No such file or directory"

Any idea ? I looked here and here but couldn't make it work.

Thank you for your help !

Post the entire stack trace. Also be careful where the scripts are located, both the python and R ones, for easiness you could place them all in the same directory. — gplayer
– gplayer, Commented Apr 28, 2016 at 8:59

John Zwinck · Accepted Answer · 2016-04-28 09:13:27Z

So what "module load" actually does is set some environment variables in the calling shell. So when you do this:

os.system('module load R/3.2.3')

Python creates a process, runs /bin/sh in it, and passes that command to the shell. The module environment variables are set in that shell. Then that shell exits--job done!

The environment variables do not--and cannot--propagate back to the Python process. So when you do this:

sp.check_output(['Rscript','test.R'])

It's totally irrelevant that you ran module load before.

So how can you fix this? Well, one possibility would be to explicitly specify the path to Rscript:

sp.check_output(['/your/full/path/to/Rscript','test.R'])

Another would be to combine your commands:

sp.check_output('module load R/3.2.3 && Rscript test.R', shell=True)

Finally, you could simply run module load before running your Python script in the first place. The environment variables it sets can propagate all the way to the R invocation within Python.

By the way, it is possible to invoke R directly from Python: http://rpy.sourceforge.net/rpy2/doc-dev/html/introduction.html

Collectives™ on Stack Overflow

Using python subprocess with module load

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related