How to programmatically replace PYTHONPATH

Question

Is there a way to programmatically replace PYTHONPATH ? I should add that I want to run decode.py with spark-submit

I have the following directory structure at some-path:

decode.py
decode2.py
crfsuite/
    crfsuite.py
    _crfsuite.so
    libcqdb-0.12.so
    libcrfsuite-0.12.so

decode.py:

import crfsuite
if __name__ == '__main__':
    tagger = crfsuite.Tagger()

The following command works:

PYTHONPATH=./crfsuite LD_LIBRARY_PATH=./crfsuite python decode.py

Or if I copied crfsuite/crfsuite.py and crfsuite/_crfsuite.so to my local directory (where decode.py exists), then the following also works:

LD_LIBRARY_PATH=./crfsuite python decode.py

Is there a way to programmatically add .py and .so files? I came up with decode2.py:

from ctypes import *
import imp

if __name__ == '__main__':
    cdll.LoadLibrary('<some-path>/crfsuite/_crfsuite.so')
    imp.load_source('crfsuite', '<some-path>/crfsuite/crfsuite.py')
    tagger = crfsuite.Tagger()

Executing:

LD_LIBRARY_PATH=./crfsuite python decode2.py                                

Traceback (most recent call last):
  File "decode2.py", line 6, in <module>
    imp.load_source('crfsuite', '<some-path>/crfsuite/crfsuite.py')
  File "<some-path>/crfsuite/crfsuite.py", line 17, in <module>
    _crfsuite = swig_import_helper()
  File "<some-path>/crfsuite/crfsuite.py", line 16, in swig_import_helper
    return importlib.import_module('_crfsuite')
  File "sw/anaconda2/lib/python2.7/importli/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named _crfsuite

@Jean-FrançoisFabre it would be append or insert. sys.path is not a set, it is a list. — Open AI - Opting Out
– Open AI - Opting Out, Commented Apr 5, 2017 at 21:35
Ah, I should add that I want to run decode.py with spark-submit . Thus the motivation for doing away with PYTHONPATH — ysc
– ysc, Commented Apr 5, 2017 at 21:35
@PeterWood sorry I meant append of course. sys.path.append('<some-path>/crfsuite') — Jean-François Fabre
– Jean-François Fabre ♦, Commented Apr 5, 2017 at 22:05
@Jean-FrançoisFabre , would you like to rewrite your comment as an answer? I can then check it as the accepted answer. — ysc
– ysc, Commented Apr 9, 2017 at 15:30

Jean-François Fabre · Accepted Answer · 2017-04-09 16:35:09Z

3

PYTHONPATH env. variable contents is copied in sys.path module, so before importing you can add some paths like this for instance:

import sys
sys.path.append("/path/to/your/module")
# your_module.py will now be found in /path/to/your/module
import your_module

note that you can also remove paths using sys.path.remove if PYTHONPATH overrides some system lib and you don't want it to.

answered Apr 9, 2017 at 16:35

Jean-François Fabre♦

141k24 gold badges179 silver badges246 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to programmatically replace PYTHONPATH

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related