727

I'm writing a Python application that takes a command as an argument, for example:

$ python myapp.py command1

I want the application to be extensible, that is, to be able to add new modules that implement new commands without having to change the main application source. The tree looks something like:

myapp/
    __init__.py
    commands/
        __init__.py
        command1.py
        command2.py
    foo.py
    bar.py

So I want the application to find the available command modules at runtime and execute the appropriate one.

Python defines an __import__() function, which takes a string for a module name:

__import__(name, globals=None, locals=None, fromlist=(), level=0)

The function imports the module name, potentially using the given globals and locals to determine how to interpret the name in a package context. The fromlist gives the names of objects or submodules that should be imported from the module given by name.

Source: https://docs.python.org/3/library/functions.html#__import__

So currently I have something like:

command = sys.argv[1]
try:
    command_module = __import__("myapp.commands.%s" % command, fromlist=["myapp.commands"])
except ImportError:
    # Display error message

command_module.run()

This works just fine, I'm just wondering if there is possibly a more idiomatic way to accomplish what we are doing with this code.

Note that I specifically don't want to get in to using eggs or extension points. This is not an open-source project and I don't expect there to be "plugins". The point is to simplify the main application code and remove the need to modify it each time a new command module is added.

See also: How can I import a module dynamically given the full path?

Improve this question
5
  • What does the fromlist=["myapp.commands"] do? Commented Jun 6, 2012 at 10:29
  • 2
    @PieterMüller : in a python shell type this: dir(__import__). The fromlist should be a list of names to emulate "from name import ...". Commented Aug 9, 2012 at 7:13
  • 4
    Import module from string variable Commented Aug 12, 2015 at 18:28
  • 1
    As of 2019, you should look for importlib: stackoverflow.com/a/54956419/687896 Commented Mar 13, 2019 at 9:42
  • 1
    Do not use __import__ see Python Doc use a importlib.import_module Commented Jun 28, 2019 at 15:00

10 Answers 10

Reset to default
438

The recommended way for Python 2.7 and 3.1 and later is to use importlib module:

importlib.import_module(name, package=None)

Import a module. The name argument specifies what module to import in absolute or relative terms (e.g. either pkg.mod or ..mod). If the name is specified in relative terms, then the package argument must be set to the name of the package which is to act as the anchor for resolving the package name (e.g. import_module('..mod', 'pkg.subpkg') will import pkg.mod).

e.g.

my_module = importlib.import_module('os.path')
Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

Recommended by which source or authority?
Documentation advises against using __import__ function in favor of above mentioned module.
This works well for import os.path ; how about from os.path import *?
I get the answer here stackoverflow.com/a/44492879/248616 ie. calling globals().update(my_module.__dict)
@NamGVU, this is a dangerous method since it's polluting your globals and can override any identifiers with the same names. The link you've posted has a better, improved version of this code.
|
421

With Python older than 2.7/3.1, that's pretty much how you do it.

For newer versions, see importlib.import_module for Python 2 and Python 3.

Or using __import__ you can import a list of modules by doing this:

>>> moduleNames = ['sys', 'os', 're', 'unittest'] 
>>> moduleNames
['sys', 'os', 're', 'unittest']
>>> modules = map(__import__, moduleNames)

Ripped straight from Dive Into Python.

Improve this answer

11 Comments

what is the differece to exec?
One problem with this solution for the OP is that trapping exceptions for one or two bad "command" modules makes his/her whole command-app fail on one exception. Personally I'd for loop over each import individually wrapped in a try: mods=__import__()\nexcept ImportError as error: report(error) to allow other commands to continue to work while the bad ones get fixed.
Another problem with this solution, as Denis Malinovsky points out below, is that the python docs themselves recommend not using __import__. The 2.7 docs: "Because this function is meant for use by the Python interpreter and not for general use it is better to use importlib.import_module()..." When using python3, the imp module solves this proble, as monkut mentions below.
@JohnWu why do you need foreach when you have for element in and map() though :)
Note that the map will NOT work in python 3, since map now uses lazy evaluation.
|
139

Note: imp is deprecated since Python 3.4 in favor of importlib.

As mentioned the imp module provides you loading functions:

imp.load_source(name, path)
imp.load_compiled(name, path)

I've used these before to perform something similar.

In my case I defined a specific class with defined methods that were required. Once I loaded the module I would check if the class was in the module, and then create an instance of that class, something like this:

import imp
import os

def load_from_file(filepath):
    class_inst = None
    expected_class = 'MyClass'

    mod_name,file_ext = os.path.splitext(os.path.split(filepath)[-1])

    if file_ext.lower() == '.py':
        py_mod = imp.load_source(mod_name, filepath)

    elif file_ext.lower() == '.pyc':
        py_mod = imp.load_compiled(mod_name, filepath)

    if hasattr(py_mod, expected_class):
        class_inst = getattr(py_mod, expected_class)()

    return class_inst
Improve this answer

4 Comments

Good and simple solution. I'we written a similar one: stamat.wordpress.com/dynamic-module-import-in-python But your's has some flaws: What about exceptions? IOError and ImportError? Why not check for the compiled version first and then for the source version. Expecting a class reduces reusability in your case.
In the line where you construct MyClass in the target module you are adding a redundant reference to the class name. It is already stored in expected_class so you could do class_inst = getattr(py_mod,expected_name)() instead.
Note that if a properly matching byte-compiled file (with suffix .pyc or .pyo) exists, it will be used instead of parsing the given source file. https://docs.python.org/2/library/imp.html#imp.load_source
Heads up: this solution works; however, the imp module is going to be deprecated in favor of import lib, see imp's page: """Deprecated since version 3.4: The imp package is pending deprecation in favor of importlib."""
57

Using importlib

Importing a source file

Here is a slightly adapted example from the documentation:

import sys
import importlib.util

file_path = 'pluginX.py'
module_name = 'pluginX'

spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)

# Verify contents of the module:
print(dir(module))

From here, module will be a module object representing the pluginX module (the same thing that would be assigned to pluginX by doing import pluginX). Thus, to call e.g. a hello function (with no parameters) defined in pluginX, use module.hello().

To get the effect "importing" functionality from the module instead, store it in the in-memory cache of loaded modules, and then do the corresponding from import:

sys.modules[module_name] = module

from pluginX import hello
hello()

Importing a package

To import a package instead, calling import_module is sufficient. Suppose there is a package folder pluginX in the current working directory; then just do

import importlib

pkg = importlib.import_module('pluginX')

# check if it's all there..
print(dir(pkg))
Improve this answer

4 Comments

add sys.modules[module_name] = module to the end of your code if you want to be able to import module_name afterwards
I used this in a test that needed to calculate the pathname of a script, then import it so that unit tests could be run against the functions in the script. One slight wrinkle was that I needed to rename my target script to have a .py extension.
Hey @GrantMcLean, thank for the feedback. When you say "I needed to rename the target script to have a .py extension", what do you mean exactly? I.e, did you have to append .py to the module name, or where had you have to rename?
@Brandt I use a .py extemsion on Python module files but I tend not to use an extension at all for Python scripts (i.e. the name of the file I need to type to run the script from the command-line). I found the code snippet in this answer only works if the file_path variable points to a file with a .py extension.
20

Use the imp module, or the more direct __import__() function.

Improve this answer

1 Comment

Do not use __import__ see Python Doc use a importlib.import_module
14

If you want it in your locals:

>>> mod = 'sys'
>>> locals()['my_module'] = __import__(mod)
>>> my_module.version
'2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)]'

same would work with globals()

Improve this answer

2 Comments

The documentation for the built-in locals() function explicitly warns that the "contents of this dictionary should not be modified".
This doesn't add anything useful over previously existing answers; once the module has been imported (whether by the import statement, __import__ builtin, an importlib library wrapper or anything else), it is a module object, and tasks like passing that object around, giving it a name, etc. are not special.
14

You can use exec:

exec("import myapp.commands.%s" % command)
Improve this answer

5 Comments

How do I get a handle to the module through the exec, so that I can call (in this example) the .run() method?
You can then do getattr(myapp.commands, command) to access the module.
... or add as command_module to the end of import statement and then do command_module.run()
In some version of Python 3+ exec was converted into a function with the added benefit that the resultant referenced created in source get stored into the locals() argument of exec(). To further isolate the exec'd referenced from the local code block locals you can provide your own dict, like an empty one and reference the references using that dict, or pass that dict to other functions exec(source, gobals(), command1_dict) .... print(command1_dict['somevarible'])
Do not use exec or eval on data that could ever possibly come, in whole or in part, from outside of the program. It is a critical security risk. By doing this, you allow the creator of the data to run arbitrary code on your computer. It cannot easily be sandboxed, and ordinary attempts at sandboxes for these tools are usually still very vulnerable.
2

Similar as @monkut 's solution but reusable and error tolerant described here http://stamat.wordpress.com/dynamic-module-import-in-python/:

import os
import imp

def importFromURI(uri, absl):
    mod = None
    if not absl:
        uri = os.path.normpath(os.path.join(os.path.dirname(__file__), uri))
    path, fname = os.path.split(uri)
    mname, ext = os.path.splitext(fname)

    if os.path.exists(os.path.join(path,mname)+'.pyc'):
        try:
            return imp.load_compiled(mname, uri)
        except:
            pass
    if os.path.exists(os.path.join(path,mname)+'.py'):
        try:
            return imp.load_source(mname, uri)
        except:
            pass

    return mod
Improve this answer

Comments

0

The below piece worked for me:

>>>import imp; 
>>>fp, pathname, description = imp.find_module("/home/test_module"); 
>>>test_module = imp.load_module("test_module", fp, pathname, description);
>>>print test_module.print_hello();

if you want to import in shell-script:

python -c '<above entire code in one line>'
Improve this answer

Comments

-9

The following worked for me:

import sys, glob
sys.path.append('/home/marc/python/importtest/modus')
fl = glob.glob('modus/*.py')
modulist = []
adapters=[]
for i in range(len(fl)):
    fl[i] = fl[i].split('/')[1]
    fl[i] = fl[i][0:(len(fl[i])-3)]
    modulist.append(getattr(__import__(fl[i]),fl[i]))
    adapters.append(modulist[i]())

It loads modules from the folder 'modus'. The modules have a single class with the same name as the module name. E.g. the file modus/modu1.py contains:

class modu1():
    def __init__(self):
        self.x=1
        print self.x

The result is a list of dynamically loaded classes "adapters".

Improve this answer

3 Comments

Please don't bury answers without providing a reason in the comments. It doesn't help anyone.
This is bad not just because it's bad python, but also it's just generally bad code. What is fl? The for loop definition is overly complex. The path is hard coded (and for an example, irrelevant). It's using discouraged python (__import__), what's all that fl[i] stuff for? This is basically unreadable, and is unnecessarily complex for something that is not all that hard - see the top voted answer with its one-liner.
This is bad because it doesn't engage with the question as asked, and instead invents a bunch of new requirements to address.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.