Python Plugin-System - HOWTO

Do not walk the filesystem (!) and scan the Python source code of the backends! That's an ugly hack at the best of times, and even worse here because you don't need anything like it at all! Registering all the classes on import is perfectly OK.

Store the backends in a class attribute instead of an instance attribute; that way, all Storage instances will look at the same set of backends:

>>> class Storage(object):
...     backends = set()
...
...     def register(self, backend):
...             self.backends.add(backend)
...

Every backend can register itself by instantiating its own Storage, which has access to the class-level backends attribute:

>>> foo = Storage()
>>> foo.register("text")
>>> bar = Storage()
>>> bar.register("xml")

You can read this attribute by instantiating another Storage, which will read the same variable:

>>> baz = Storage()
>>> baz.backends
{'xml', 'text'}

You could even store the backend instances in a class attribute of Connection, and register each backend upon instantiation:

>>> class Connection(object,metaclass=abc.ABCMeta):
...     @abc.abstractmethod
...     def register(self, backend):
...             pass
...
...     backends = set()
...
>>> class TextBackend(Connection):
...     def register(self):
...             super().backends.add(self)
...
...     def __init__(self):
...             self.register()
...
>>> class XMLBackend(Connection):
...     def register(self):
...             super().backends.add(self)
...
...     def __init__(self):
...             self.register()
...
>>> foo = TextBackend()
>>> bar = XMLBackend()
>>> Connection.backends
{<__main__.XMLBackend object at 0x027ADAB0>, \
<__main__.TextBackend object at 0x027ADA50>}

If these backends are going to be distributed in various Python distributions, you might want to look at setuptools/distribute entry points. Here's an article on how you might use these for dynamic plugin finding services:

http://aroberge.blogspot.com/2008/12/plugins-part-6-setuptools-based.html

But should your application always write to all backends? If not, you could use (as usual) another layer of indirection, e.g.

storage = Storage()
storage.use(TextBackend, XMLBackend, YamlBackend)
storage.write(data)

Or something like this, with Storage being a dispatcher, which would just loop over the backends and call the appropriate serializer.

This is of course, very coarse.

you could use a function like this one:

def loadClass(fullclassname):
    sepindex=fullclassname.rindex('.')    
    classname=fullclassname[sepindex+1:]
    modname=fullclassname[:sepindex]
    #dynmically import the class in the module
    imod=__import__(modname,None,None,classname)
    classtype=getattr(imod,classname)
    return classtype

where fullclassname is the fully dotted qualifiant for the class you want load.

example (pseudo code,but the idea is there):

for package availability scanning, only perform some globbing , then for finding final class name, you may declare a Plugin class in each of your modules that has a getStorage()

 #scan for modules , getPluginPackagesUnder (to be defined) returns the dotted name for all packages under a root path (using some globbing, listdir or whatever method) 

    pluginpackages=getPluginPackagesUnder("x/y/z")
    storagelist=[]
    for pgpck in plunginpackages:
        pluginclass=loadClass("%s.Plugin"%pgpck)
        storageinstance=Plugin().getStorage()
        storagelist.append(storageinstance)

so, you can dynamically scan for your existing storage plugins