Include python module (dependencies) installation in your python script

Question

Is there a way to include/invoke python module(s) (dependencies) installation first, before running the actual/main script?

For example, in my main.py:

import os, sys
import MultipartPostHandler

def main():
    # do stuff here

But MultipartPostHandler is not yet installed, so what I want is to have it installed first before actually running main.py... but in an automated manner. When I say automatically, I mean I will just invoke the script one time to start the dependency installation, then to be followed by actual functionalities of the main script. (somehow, a little bit similar with maven. But I just need the installation part)

I already know the basics of setuptools. The problem is I may have to call the installation (setup.py) and the main script (main.py) separately.

Any ideas are greatly appreciated. Thanks in advance!

What is the python version? python 2 or 3?

Anthony Kong
– Anthony Kong

2014-09-28 05:30:45 +00:00
Commented Sep 28, 2014 at 5:30 — Anthony Kong
– Anthony Kong, Commented Sep 28, 2014 at 5:30
Thanks for the reply. It's python 2.7 Sir

jaysonpryde
– jaysonpryde

2014-09-28 05:49:01 +00:00
Commented Sep 28, 2014 at 5:49 — jaysonpryde
– jaysonpryde, Commented Sep 28, 2014 at 5:49

dnozay · Accepted Answer · 2020-03-12 00:03:22Z

9

Is there a way to include/invoke python module(s) (dependencies) installation first, before running the actual/main script?

A good way is to use setuptools and explicitly list them in install_requires.
Since you are providing a main function, you also probably want to provide entry_points.

I already know the basics of setuptools. The problem is I may have to call the installation (setup.py) and the main script (main.py) separately.

That is usually not a problem. It is very common to first install everything with a requirements.txt file and pip install -r requirements.txt. Plus if you list dependencies you can then have reasonable expectations that it will be there when your function is called and not rely on try/except ImporError. It is a reasonable approach to expect required dependencies to be present and only use try/except for optional dependencies.

setuptools 101:

create a tree structure like this:

$ tree
.
├── mymodule
│   ├── __init__.py
│   └── script.py
└── setup.py

your code will go under mymodule; let's imagine some code that does a simple task:

# module/script.py    

def main():
    try:
        import requests
        print 'requests is present. kudos!'
    except ImportError:
        raise RuntimeError('how the heck did you install this?')

and here is a relevant setup:

# setup.py

from setuptools import setup
setup(
    name='mymodule',
    packages=['mymodule'],
    entry_points={
        'console_scripts' : [
            'mycommand = mymodule.script:main',
        ]
    },
    install_requires=[
        'requests',
    ]
)

This would make your main available as a command, and this would also take care of installing the dependencies you need (e.g requests)

~tmp damien$ virtualenv test && source test/bin/activate && pip install mymodule/
New python executable in test/bin/python
Installing setuptools, pip...done.
Unpacking ./mymodule
  Running setup.py (path:/var/folders/cs/nw44s66532x_rdln_cjbkmpm000lk_/T/pip-9uKQFC-build/setup.py) egg_info for package from file:///tmp/mymodule

Downloading/unpacking requests (from mymodule==0.0.0)
  Using download cache from /Users/damien/.pip_download_cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2F2.7%2Fr%2Frequests%2Frequests-2.4.1-py2.py3-none-any.whl
Installing collected packages: requests, mymodule
  Running setup.py install for mymodule

    Installing mycommand script to /tmp/test/bin
Successfully installed requests mymodule
Cleaning up...
(test)~tmp damien$ mycommand 
requests is present. kudos!

more useful commands with argparse:

If you want to use argparse then...

# module/script.py

import argparse

def foobar(args):
    # ...

def main():
    parser = argparse.ArgumentParser()
    # parser.add_argument(...)
    args = parser.parse_args()
    foobar(args)

edited Mar 12, 2020 at 0:03

answered Sep 28, 2014 at 7:24

dnozay

24.4k6 gold badges85 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

jaysonpryde Over a year ago

thank you very much for this comprehensive answer. greatly appreciate it dnozay

jaysonpryde Over a year ago

question though, what if my main expects arguments from argparse? Do I have to specify something under entry_points? Thanks in advance!

dnozay Over a year ago

you would list argparse as a dependency or check for python >= 2.7, then in your main function, build the parser and call it. Instead of putting the code somewhere in the body of the module after e.g. if __name__ == '__main__': you would put it in a function that is exposed by console_scripts.

jaysonpryde Over a year ago

thanks a lot again. But hopefully, can you show snippet codes? If not too much to ask. Thanks

mTheSame Over a year ago

For those who are looking for the referenced links, can be found here: setuptools.readthedocs.io/en/latest/…

|

RevolutionTech · Accepted Answer · 2014-09-28 07:06:04Z

There's a few ways to do this. One way is to surround the import statement with a try...except ImportError block and then have some Python code that installs the package if the ImportError exception is raised, so something like:

try:
    import MultipartPostHandler
except ImportError:
    # code that installs MultipartPostHandler and then imports it

I don't think this approach is very clean. Plus if there are other unrelated importing issues, that won't be detected here. A better approach might be to have a bash script that checks to see if the module is installed:

pip freeze | grep MultipartPostHandler

and if not, installs the module:

pip install MultipartPostHandler

Then we can safely run the original Python code.

EDIT: Actually, I like FLORET's answer better. The imp module is exactly what you want.

Régis FLORET · Accepted Answer · 2014-09-28 07:03:25Z

-2

You should use the imp module. Here's a example:

import imp
import httplib2
import sys

try:
  import MultipartPostHandler
except ImportError:
  # Here you download 
  http = httplib2.Http()
  response, content = http.request('http://where_your_file_is.com/here')
  if response.status == 200:
    # Don't forget the right managment
    with open('MultipartPostHandler.py', 'w') as f:
     f.write(content)
    file, pathname, description = imp.find_module('MultipartPostHandler')
    MultipartPostHandler = imp.load_module('MultipartPostHandler', file, pathname, description)
  else:
    sys.exit('Unable to download the file')

For a full approach, use a queue:

download_list = []
try:
    import FirstModule
except ImportError:
    download_list.append('FirstModule')

try:
    import SecondModule
except ImportError:
    download_list.append('SecondModule')

if download_list:
    # Here do the routine to dowload, install and load_modules

# THe main routine
def main():
    the_response_is(42)

You can download binaries with open(file_content, 'wb')

I hope it help

BR

answered Sep 28, 2014 at 7:03

Régis FLORET

4942 silver badges9 bronze badges

2 Comments

dnozay Over a year ago

this answer doesn't explain why it is a good approach. I certainly wouldn't want my scripts to start downloading extra data and imports off the internet while they are running.

Régis FLORET Over a year ago

@dnozay I agree, I'll never write something like that mainly for security reasons ;) But sometimes, there is a little thing called reality, you know day work, bosses, clients... The good response is when the code works (my response). The best response is yours.

Collectives™ on Stack Overflow

Include python module (dependencies) installation in your python script

3 Answers 3

setuptools 101:

more useful commands with argparse:

6 Comments

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

setuptools 101:

more useful commands with argparse:

6 Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related