Add a subdirectory to a python namespace

Question

I would like to be able to import a python module which is actually located in a subdirectory of another module.

I am developing a framework with plug-ins. Since I'm expecting to have a few thousands (there's currently >250 already) and I don't want one big directory containing >1000 files I have them ordered in directories like this, where they are grouped by the first letter of their name:

framework\
    __init__.py
    framework.py
    tools.py
    plugins\
        __init__.py
        a\
            __init__.py
            atlas.py
            ...
        b\
            __init__.py
            binary.py
            ...
        c\
            __init__.py
            cmake.py
        ...

Since I would not like to impose a burden on developers of other plugins, or people not needing as many as I have, I would like to put each plugin in the 'framework.plugins' namespace. This way someone adding a bunch of private plugins can just do so by adding them in the folder framework.plugins and there provide a __init__.py file containing:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

however, currently this setup is forcing them to also use the a-z subdirectories. Sometimes a plugin is extending another plugin, so now I have a

from framework.plugins.a import atlas

and I would like to have

from framework.pugins import atlas

Is there any way to declare a namespace where the full name space name actually doesn't map to a folder structure?

I am aware of the pkg_resources package, but this is only available via setuptools, and I'd rather not have an extra dependency.

import pkg_resources
pkg_resources.declare_namespace(__name__)

The solution should work in python 2.4-2.7.3

update: Combining the provided answers I tried to get a list of all plugins imported in the __init__.py from plugins. However, this fails due to dependencies. Since a plugin in the 'c' folder tries to import a plugin starting with 't', and this one has not been added yet.

plugins = [ x[0].find_module(x[1]).load_module(x[1]) for x in pkgutil.walk_packages([ os.path.join(framework.plugins.__path__[0], chr(y)) for y in xrange(ord('a'), ord('z') + 1) ],'framework.plugins.' ) ]

I'm not sure If I'm on the right track here, or just overcomplicating things and better write my own PEP302 importer. However, I can't seem to find any decent examples of how these should work.

Update: I tried to follow the suggesting of wrapping the __getattr__ function in my __init__.py, but this seems to no avail.

import pkgutil
import os
import sys
plugins = [x[1] for x in pkgutil.walk_packages([ os.path.join(__path__[0], chr(y)) for y in xrange(ord('a'), ord('z') + 1) ] )]
import types
class MyWrapper(types.ModuleType):
    def __init__(self, wrapped):
            self.wrapped = wrapped

    def __getattr__(self, name):
           if name in plugins:
                   askedattr =  name[0] + '.' + name
            else:
                    askedattr = name
            attr = getattr(self.wrapped, askedattr)
            return attr


sys.modules[__name__] = MyWrapper(sys.modules[__name__])

Out of curiosity, do you have a specific reason for which you'd rather not depend on setuptools? Unless you're in a very specific situation, I honestly think you can expect reasonnably expect users to have setuptools / pip installed anyway (And setuptools itself is widely available and easily installed using the egg.). — Thomas Orozco
– Thomas Orozco, Commented Aug 20, 2012 at 16:34
This is a framework for compiling software from source currently counting >16000 lines of code. We don't assume the user has root privileges and/or can easily install software in any other way. So currently the only real dependency is python> 2.4 < 3.0. — Jens Timmerman
– Jens Timmerman, Commented Aug 20, 2012 at 16:54
setuptools/pip will either install to the root installation or to the user's home directory ( or to a virtualenv ). it's not necessary to have root privileges to install python modules. — Jonathan Vanasco
– Jonathan Vanasco, Commented Aug 21, 2012 at 16:27
Yes, but if you haven't got setuptools or pip yet, it's not that easy to actually install it. We're going for 1 tarbal, unzipping it, done. Also, I felt there had to be a really simple way, I found it with manually extending __path__ now. So really not needed. Also, declare_namespace still needed you to leave the directory structure to agree with the namespace. — Jens Timmerman
– Jens Timmerman, Commented Aug 23, 2012 at 8:56

Pierre GM · Accepted Answer · 2012-08-20 15:53:43Z

1

A simple solution would be to import your a, b... modules in plugins.__init__, like:

from a import atlas
from b import binary
...

answered Aug 20, 2012 at 15:53

Pierre GM

20.5k3 gold badges58 silver badges67 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Jens Timmerman Over a year ago

I would be adding new plugins all the day, since this framework is under heavy development by a couple of different developers, adding an import every time is just not an option. I'm looking into a clean solution, preferably only 1-2 lines of python.

Pierre GM Over a year ago

And what if, say, you have another atlas.py in module b ?

Jens Timmerman Over a year ago

I'm sorry, the a-z directory structure is there to reflect the first letter of the module name, this is to group the files, so I don't end up with 1 directory containing >1000 files.

Pierre GM Over a year ago

Then, what about pkgutil.extend_path, as described here and [there] (docs.python.org/library/pkgutil.html) ?

Jonathan Vanasco Over a year ago

why don't you want a directory with > 1000 files ? any modern filesystem can handle that number fine. you're also not really safeguarding against any real folder content limits - this assumes there's a relatively even distribution of modules by the first letter - this rarely/never happens. it's not unlikely that a single folder could have over 1000 modules if you double your current size.

|

ecatmur · Accepted Answer · 2012-08-21 09:19:57Z

1

I'd suggest lazy-loading the modules by subclassing types.ModuleType and converting modname to modname[0] + '.' + modname on demand; this should be less work than implementing a module loader.

You can look at apipkg for an example of how to do this.

answered Aug 21, 2012 at 9:19

ecatmur

158k28 gold badges311 silver badges387 bronze badges

7 Comments

Jens Timmerman Over a year ago

How would I then let python know to use this new ModuleType for modules in the framework.plugins directory?

ecatmur Over a year ago

@JensTimmerman you replace the module in sys.modules[pkgname]. See bitbucket.org/hpk42/apipkg/src/tip/apipkg.py#cl-14

Jens Timmerman Over a year ago

So I had to repeat this for every module again? This will again give me the dependency problem where I have to first resolve the dependencies. I'd rather just tell python to use the new type for each module in this submodule, but assigning to type.ModuleType does not work...

ecatmur Over a year ago

@JensTimmerman no, just for the module framework.plugins; a reference from framework.plugins modname is a getattr on sys.modules["framework.plugins"].

Jens Timmerman Over a year ago

Yes, I meant this basically is about same as importing each plugin in the framework/plugins/__init__.py ? But doing it lazy.

|

Jens Timmerman · Accepted Answer · 2012-10-15 14:52:33Z

1

Don't use the pkgutil.extend_path function here, it tries to do the opposite of what you're trying to accomplish:

This will add to the package’s __path__ all subdirectories of directories on sys.path named after the package. This is useful if one wants to distribute different parts of a single logical package as multiple directories.

Just extending __path__ with the subdirectories in your framework.plugins.__init__.py works just fine.

So the solution to this problem is: put this in your __init__.py:

__path__.extend([os.path.join(__path__[0],chr(y)) for y in range(ord('a'),ord('z')+1)])

edited Oct 15, 2012 at 14:52

answered Aug 21, 2012 at 16:10

Jens Timmerman

9,9912 gold badges45 silver badges50 bronze badges

Comments

nOw2 · Accepted Answer · 2012-08-20 16:26:04Z

0

This isn't a particularly fast solution (startup overheads), but what about having plugins.__init__ scrape the filesystem and import each found file into the local namespace?

import glob
import sys

thismodule = sys.modules[__name__]

for plugin in glob.glob("?/*"):
    _temp = __import__(plugin.split("/")[0],
                   globals(),
                   locals(),
                   [plugin.split("/")[1]],
                   -1)
   setattr(thismodule, plugin.split("/")[1], getattr(_temp, plugin.split("/")[1]))

answered Aug 20, 2012 at 16:26

nOw2

6667 silver badges13 bronze badges

3 Comments

Jens Timmerman Over a year ago

Your code got me thinking, and I came up with this: modules = [ x[0].find_module(x[1]).load_module(x[1]) for x in pkgutil.walk_packages([ os.path.join(framework.plugins.__path__[0],chr(y)) for y in xrange(ord('a'),ord('z')) ],'framework.plugins.' ) ] But this doesn't work because of dependencies. A plugin in the /c/ directory already tries to import one in the /t/ directory, which is not available yet at that time, since I haven't added it as an attribute yet.

nOw2 Over a year ago

Interdependence like that could cause you more problems further on - either import directly (from plugins.t import Thing) or rework the interdependencies to something external to the plugins.

Jens Timmerman Over a year ago

We're currently using the direct import indeed. But I thought it would be relatively easy to just add everything to a namespace, without the directory structure interfering. I just can't seem to find the easiest way.

Collectives™ on Stack Overflow

Add a subdirectory to a python namespace

4 Answers 4

7 Comments

7 Comments

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

7 Comments

7 Comments

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related