specify the python interpreter path in python script

Question

I have a python script which I am distributing in several nodes. I have python 2.6 installed in /usr/bin by default and have python 2.7 in my /opt directory in all the nodes. Now when I run the script from my current node I can set the path to python 2.7 interpreter from terminal but I am unable to manage it in the rest of the nodes where this script is getting distributed. I have added the shebang at the start of script like -

#!/opt/python2.7/bin/python

But its still not working. How can I change the python interpreter/python path at the beginning of the script itself.

This should work. What is the error you are getting? Is the script executable? — deets
– deets, Commented Oct 19, 2017 at 15:59
It is still pointing to the default python path. So its unable to identify the import packages. — Y0gesh Gupta
– Y0gesh Gupta, Commented Oct 19, 2017 at 16:02
What does "print sys.prefix" give you? How do you invoke the script? — deets
– deets, Commented Oct 19, 2017 at 16:03
Your issue isn't the interpreter being used, potentially, but your PATH — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Oct 19, 2017 at 16:06
Yes, so I dont have any mean to update my python path in rest of the nodes. In rest of the nodes the python path is set to python 2.6. Is there a way to update the python path through my python script? — Y0gesh Gupta
– Y0gesh Gupta, Commented Oct 19, 2017 at 16:08

nudomarinero · Accepted Answer · 2017-10-19 16:46:03Z

3

What you explain should work but check:

that the script is executable (chmod +x my_script.py if required).
that you are calling the script directly and not using another Python interpreter (check that you execute ./my_script.py or /path/my_script.py and not python my_script.py).

To help to diagnose the problem you could add the following lines to the top of your script:

#!/opt/python2.7/bin/python
import sys
print(sys.executable)

if the output is not /opt/python2.7/bin/python you might be calling the script with another interpreter.

If for some reason you can only call scripts executed by the 2.6 version of Python remotely but you can also distribute additional files, you could try to send your main script somewhere and execute the following auxiliary script:

from subprocess import call
call("/opt/python2.7/bin/python /path/my_scipt.py", shell=True)

edited Oct 19, 2017 at 16:46

answered Oct 19, 2017 at 16:16

nudomarinero

1063 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Y0gesh Gupta Over a year ago

yes the issue is the default interpreter in rest of the machines is python 2.6 when the python job gets distributed its using that. How can I make the job point to the 2.7 interpreter in the code itself.

nudomarinero Over a year ago

As far as I know you cannot but there may be other ways to solve your problem. Two questions: can you only execute Python 2.6 remotely or can you execute files in a shell? Can you distribute two scripts instead of just one in the remote nodes?

Y0gesh Gupta Over a year ago

I am distributing my script as a hive udf, so my script is disributed across all the nodes and is executed by hive. What I am assuming is hive runs it using default interpreter in each node which points to python 2.6 by default. So although we have python 2.7 installed but since its not in the path we cant execute using it by default.

nudomarinero Over a year ago

Then your problem is more related to Hive and the way it handles UDFs. I do not have much experience with Hive but I would try to add the script (with the shebang) with something like ADD FILE hdfs:///tmp/my_script.py; USE tmp; and then either: * call it using something like ... USING 'my_script.py' ..., or * call it specifying the full path to Python ... USING '/opt/python2.7/bin/python my_script.py' ...

Collectives™ on Stack Overflow

specify the python interpreter path in python script

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related