1

My system administrator will not allow global installation of python packages. I'm writing a script that people will invoke to perform certain actions for them. The script I'm writing needs certain libraries like sqlalchemy and coloredlogs. I am however allowed to install python libs any local folder. i.e not site-packages.

How would I go about installing the libs in the same folder as the script so that the script has access to them?

My folder hierarchy is like so

script_to_invoke.py
scriptpack/
    bin/
        coloredlogs
        coloredlogs.egg
        ...
    utils/
        util1.py
        util2.py

(all the folders indicated have an __init__.py)

What I've tried so far:

within script_to_invoke.py I use

from scriptpack.utils invoke util1 # no problem here
from scriptpack.bin import coloredlogs # fails to find the import

I've looked at some other SO answers abut I'm not sure how to correlate them with my problem.

Improve this question
8
  • There is no coloredlogs.py but coloredlogs.egg Commented Mar 7, 2018 at 0:36
  • yes, when I ran easy_install and installed coloredlogs to the bin folder, it installed only a coloredlogs and a coloredlogs.egg file . Commented Mar 7, 2018 at 0:38
  • So that's why from scriptpack.bin import coloredlogs fails Commented Mar 7, 2018 at 0:39
  • It's looking for a .py Commented Mar 7, 2018 at 0:39
  • ok, so how would I fix that? easy_install doesn't seem to put a .py file in the folder. changing coloredlogs to coloredlogs.py Commented Mar 7, 2018 at 0:43

3 Answers 3

Reset to default
2

I figured it out!

Python had to be directed to find the .egg files This can be done by either

  1. Editing the PYTHONPATH var BEFORE the interpreter is started (or)
  2. Appending the full path to the eggs to the sys path

Code Below:

import sys

for entry in [<list of full path to egg files in bin dir>]:
    sys.path.append(str(entry))

# Proceed with local imports
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

This is an incredibly brittle solution (especially the PYTHONPATH one). Among other issues, modifying PYTHONPATH affects both Python 2 and Python 3, and if the modules in question aren't 100% source compatible with both (which is rare), you'll ensure that a given module only works for one major Python version. Dedicated packaging solutions are a much better idea.
other than causing python2/3 incompatibilities , in this use case do you forsee any other problems? The target users have the path to the script and will just invoke it, no need to worry about changing paths and such.
Mostly just that it's hard to maintain compared to proper solutions. Updating egg packages in a random directory doesn't benefit much (if at all) from pip support (and requires updating in both the directory and possibly manually editing the list of eggs to load in your script), putting the support files in a user directory but making them globally accessible requires careful attention to permissions, and becomes an point of accidental failure (if you accidentally remove global read/traverse permissions from your home directory root, suddenly everything breaks), etc.
1

If you might want to try packaging up everything as a zipapp. Doing so makes a single zip file that acts as a Python script, but can contain a whole multitude of embedded packages. The steps to make it are:

  1. Make a folder with the name of your program (testapp in my example)
  2. Name your main script __main__.py and put it in that folder
  3. Using pip, install the required packages to the folder with --target=/path/to/testapp
  4. Run python3 -mzipapp testapp -p='/usr/bin/env python3' (providing the shebang line is optional; without it, users will need to run the package with python3 testapp.pyz, while with the shebag, they can just do ./testapp.pyz)

That creates a zip file with all your requirements embedded in it alongside your script, that doesn't even need to be unpacked to run (Python knows how to run zip apps natively). As a trivial example:

$ mkdir testapp
$ echo -e '#!/usr/bin/python3\nimport sqlalchemy\nprint(sqlalchemy)' > __main__.py
$ pip3 install --target=./testapp sqlalchemy
$ python3 -mzipapp testapp -p='/usr/bin/env python3'
$ ./testapp.pyz
<module 'sqlalchemy' from './testapp.pyz/sqlalchemy/__init__.py'>

showing how the simple main was able to access sqlalchemy from within the same zipapp. It's also smaller (thanks to the zipping) that distributing the uncompressed modules:

$ du -s -h testapp*
13M     testapp
8.1M    testapp.pyz
Improve this answer

4 Comments

There are more thorough solutions that don't even require the user to have Python installed (cx_freeze, py2exe, etc.), but since you seem to know they'll definitely have Python available, zipapp bundles everything besides the interpreter, which saves quite a bit on file size.
For the record, if you don't want to name your main script __main__.py, the zipapp docs cover defining an entry point (by module and function name) so you can keep your original script name, and have zipapp generate a skeleton __main__.py that invokes it.
I'll explore this, thanks!! Our system doesn't have python3, would this still work ?
@SrinivasSuresh: The zipapp module only ships with Python 3.5 and higher, but it can make zipapp's for any version of Python (the zipapp module is used to build the zip in the correct format, but isn't needed to run it). Technically, you don't even need the zipapp module at all (you can create the necessary zip file manually the same way zipapp does), but it makes it a lot simpler.
0

You can install these packages in a non-global location (generally in ~/.local/lib/python<x.y>) using the --user flag, e.g.:

pip install --user sqlalchemy coloredlogs

That way you don't have to worry about changing how imports work, and you're still compliant with your sysadmins policies.

Improve this answer

3 Comments

But people who invoke my script would not have access to the libraries?
You can add a requirements.txt file that enumerates dependencies, and in your INSTALL instructions note that the user should run pip install --user -r requirements.txt. This is a fairly standard workflow.
unfortunately asking the users to install is not an option either. I have to bundle the libs with my program :(

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.