Python C-ext namespace mixed with regular python submodules?

Question

This question has some resemblance with:

Only with a slight twist. Here I'm not trying to mix two C-exts, but one C-ext and a regular python submodule instead.

Is there a way for a C-extension to share the module namespace between the symbols "module.so" and those present in a submodule?

My module structure looks like this:

facs/
    facs/
      __init__.py
      setup.py
      facs.so
      [*.c files]
      utils/
        __init__.py
        galaxy.py

If I remove "utils" from the hierarchy, I can import facs and see the facs.so methods:

>>> import facs
>>> dir(facs)
['__doc__', '__file__', '__name__', '__package__', 'build', 'query', 'remove']

But when I put the utils submodule back and try to import the different parts, one namespace seems to mask the other (utils masks the symbols exported by facs.so):

>>> import facs
>>> dir(facs)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
>>> import facs.utils
>>> facs.utils.galaxy.rsync_genomes("phix")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'galaxy'
>>> from facs.utils import galaxy
>>> galaxy.rsync_genomes("phix")
'Hello world'

As you see, after dir(facs), build, query and remove are gone and galaxy does not get imported properly unless I do a from facs.utils import galaxy instead of re-using the initial import statement and simply accessing via facs.utils.galaxy.rsync_genomes() directly.

To sum up, my intended use of this module would be to do:

>>> import facs
>>> dir(facs)
['__doc__', '__file__', '__name__', '__package__', 'build', 'query', 'remove'
, 'utils'] <--- (Directly accessible from "facs")
>>> facs.utils.galaxy.rsync_genomes("phix")
'Hello world'

The (currently under development) code is in:

https://github.com/brainstorm/facs/tree/develop

In case someone wants to try it out by themselves. I'm using virtualenvs and my $PYTHONPATH seems correct:

/home/roman/.venvburrito/lib/python:
/home/roman/.virtualenvs/py27/lib/python2.7/site-packages

And the installation seems to be successful too:

cd ~/.virtualenvs/py27/lib/python2.7/site-packages/facs-2.0dev-py2.7.egg/
(py27)$ ls
EGG-INFO  facs.py  facs.pyc  facs.so  utils/

It seems that no __init__.py file is actually copied to the top level directory but touching it there or not does not affect the importing behavior described above.

Any ideas? Thanks in advance!

Did you try to rename facs.so to something else? Try to use a different name for the package and the .so — Bakuriu
– Bakuriu, Commented Mar 1, 2013 at 16:25
The .so file is generated when the python module is compiled/installed, I don't generate it myself... sorry if that wasn't clear :-S — brainstorm
– brainstorm, Commented Mar 1, 2013 at 17:31

brainstorm · Accepted Answer · 2013-03-04 09:11:15Z

0

Be carefull you are importing the so file first, and not the facs package. Are you running your python console over facs package source directory ?

Try to go outside the package directory and everything will make sense.

Also, next import doesn't say that all modules behind your facs directory will be imported:

>> import facs
>> dir (facs)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']

You have to set your imports in each __init__.py file, for example if you want to export some simbols from your .so file when you import facs, you should have the following contents in your __init__.py file:

>> from _facs import function_name, function_name .....

And then, when you import facs, it will import those functions for you.

Take the same pattern for your sub-package utils.

I would also recommend to rename your facs.so file to _facs.so file to avoid name collisions between package name and module name.

edited Mar 4, 2013 at 9:11

brainstorm

7607 silver badges25 bronze badges

answered Mar 1, 2013 at 17:32

pfreixes

4492 silver badges6 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

brainstorm Over a year ago

Hello pfreixes! It's nice to see you over here! I actually put in place your suggestions, but that doesn't seem to fix my problem :_S: github.com/brainstorm/facs/commit/…

brainstorm Over a year ago

After the commit above, I get "Import error: no module named facs", so I cannot load any of the symbols :-/ Again, thanks a lot for having a look at that! :D

pfreixes Over a year ago

Sorry but facs/__init__.py should be """from facs._facs import ...""" and it should be work ...

brainstorm Over a year ago

File "/home/roman/dev/facs/facs/__init__.py", line 1, in <module> from facs._facs import build, query, remove ImportError: dynamic module does not define init function (init_facs)

brainstorm Over a year ago

I've observed that the facs/__init__.py file I just edited, does not get copied to the .virtualenv dir: $ ls ~/.virtualenvs/py27/lib/python2.7/site-packages/facs-2.0dev-py2.7-linux-x86_64.egg/ EGG-INFO _facs.py _facs.pyc _facs.so utils .... but the result is the same, the module does not get imported at all (ImportError: No module named facs).

|

Collectives™ on Stack Overflow

Python C-ext namespace mixed with regular python submodules?

1 Answer 1

9 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related