How to create submodules with the Python C API?

Question

I want to create the following module and object structure:

main_module
|
`--sub_module
   |
   +--ObjectOne
   |
   `--sub_sub_module
      |
      `--ObjectTwo

When I only had main_module.sub_module everything worked as it should, as I created an empty __init__.py inside the main_module folder (since main_module does not have any objects at the moment), and I placed sub_module.so next to it.

However, when I tried to create two separate c modules to make main_module.sub_module.sub_sub_module work (that is sub_module.so and sub_sub_module.so) and then I added two __init__.pys which both of them imported their extension modules relative to them, then I got into ImportErrors (as wrappers around the undefined symbol error "raised" from the shared libraries), because sub_sub_module needs a few of the C-level definitions from sub_module. Which makes me think, it would be way more easier, to create a single shared library which somehow creates "virtual" modules inside it, instead of the unnecessary library linking..

So my question is: Is it possible? If so, how? Or, is there a better way to achieve what I'm looking for?

Could you elaborate a little on how you implemented all this; or better post some code? — tynn
– tynn, Commented Oct 10, 2015 at 19:50
@tynn it is quite large to post the whole thing here, but here is the repo: github.com/kitchenbudapest/hackathon/tree/master/fw/wrappers/… -- if you go a few folders up, in kb there is the C library I'm wrapping, and the link I provided is the python wrapper part only — Peter Varo
– Peter Varo, Commented Oct 14, 2015 at 11:09

tynn · Accepted Answer · 2015-10-15 23:57:03Z

2

I think I got your problem now. For a long and detailed answer you should read Providing a C API for an Extension Module.

Portability therefore requires not to make any assumptions about symbol visibility. This means that all symbols in extension modules should be declared static, except for the module’s initialization function, in order to avoid name clashes with other extension modules [...]. And it means that symbols that should be accessible from other extension modules must be exported in a different way.

To achieve this you should use Capsules. For these every function you want to use in another module should be stored inside a void* array. Then you create a capsule with PyCapsule_New(void*, const char*, PyCapsule_Destructor) using the array as first argument. This object must be added to your sub_module. Then you can call PyCapsule_Import(const char*, int) in your sub_sub_module to import the array and access the functions you need.

answered Oct 15, 2015 at 23:57

tynn

40k8 gold badges119 silver badges152 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Peter Varo Over a year ago

I'm already familiar with Capsules -- as You can see, I use them for the enum values (PIN indices). However I've never thought of using them like you have suggested.. My other solution -- which I think can be better than the one you have provided -- is to create 3 shared libraries: 1) for common-data, 2) for sub_module, 3) for sub_sub_module => in this case there is no "Python overhead" where it is not necessary. IMHO Capsules are there to make opaque C-level data passing by the user -- but here, the data is not necessarily used by the user directly. (+1 for the effort though!)

tynn Over a year ago

@PeterVaro it's from the official documentation. I was just paraphrasing. In my understanding Capsules should always be used instead of exporting symbols and python overhead is quite insignificant since you're only importing it once. But if it's not python related data and functions I might also link to a "pure" C library instead.

Collectives™ on Stack Overflow

How to create submodules with the Python C API?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related