Python3 reload project that use python c-api

Question

I have a project that I built for it a Class in C (python c-api) which I extended more in the python project. The project's purpose is to provide a test framework for a C library. The test is mainly executed for each pull request of the C library.

The project needs to download form a Nexus server the relevant build of the C library, Compile the python class that is dependent on the C library, then to perform the tests.

The Problem: import / reload the project modules after the compilation of C code.

The Question: In my opinion, it's not that elegant to do import in each function that depends on the C library, So I tried to invoke reload, but it seems that it doesn't work, or at least not as I expecting.

Code The code is super-simplified to illustrate the issue, you can check this thread history to see the previous code.

main.py

from utils.my_custom_py import MyCustomExtended
from importlib.util import find_spec
from importlib import reload
from os import system, stat
import weakref
import sys


def setup():
    if system('./setup.py clean build install') > 0:
        raise SystemError("Failed to setup python c-api extention class")


def main():
    if find_spec('custom2') is None:
        setup()
        for module_name in list(sys.modules.keys()):
            m = sys.modules.get(module_name)
            if not hasattr(m, '__file__'):
                continue
            if getattr(m, '__name__', None) in [None, '__mp_main__', '__main__']:
                continue

            try:
                # superreload(m)  # from ==> IPython.extensions
                # sys.modules[module_name] = reload(m)
                reload(m)
            except Exception as e:
                ...

    MyCustomExtended(1, 2, 3)
    print("COOL")


if __name__ == "__main__":
    main()

utils.my_custom_py.py

from importlib.util import find_spec

if find_spec('custom2'):
    import custom2
else:
    class custom2:
        class Custom:
            ...

class MyCustomExtended(custom2.Custom):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

setup.py

from distutils.core import Extension, setup

custom_ext = Extension("custom2", ["src/custom.c"])
setup(name="custom2", version="1.0", ext_modules=[custom_ext])

src.custom.c is taken from: docs.python.org

The error output:

running clean
running build
running build_ext
building 'custom2' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/src
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/tmp/PlayAround/.venv/include -I/usr/include/python3.6m -c src/custom.c -o build/temp.linux-x86_64-3.6/src/custom.o
creating build/lib.linux-x86_64-3.6
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/src/custom.o -o build/lib.linux-x86_64-3.6/custom2.cpython-36m-x86_64-linux-gnu.so
running install
running install_lib
copying build/lib.linux-x86_64-3.6/custom2.cpython-36m-x86_64-linux-gnu.so -> /tmp/PlayAround/.venv/lib/python3.6/site-packages
running install_egg_info
Removing /tmp/PlayAround/.venv/lib/python3.6/site-packages/custom2-1.0.egg-info
Writing /tmp/PlayAround/.venv/lib/python3.6/site-packages/custom2-1.0.egg-info
Traceback (most recent call last):
  File "./main.py", line 38, in <module>
    main()
  File "./main.py", line 33, in main
    MyCustomExtended(1, 2, 3)
  File "/tmp/PlayAround/utils/my_custom_py.py", line 12, in __init__
    super().__init__(*args, **kwargs)
TypeError: object.__init__() takes no parameters

main that work:

from importlib.util import find_spec
from importlib import reload
from os import system, stat
import weakref
import sys


def setup():
    if system('./setup.py clean build install') > 0:
        raise SystemError("Failed to setup python c-api extention class")


def main():
    if find_spec('custom2') is None:
        setup()

    from utils.my_custom_py import MyCustomExtended
    MyCustomExtended(1, 2, 3)
    print("COOL")


if __name__ == "__main__":
    main()

If your question is about reloading modules, create a script that loads and reloads a single module. — D. SM
– D. SM, Commented Nov 20, 2020 at 1:11
That is too easy, and require pre-steps, which is exactly what I'm trying to avoid :), there must be a solution with the python language for such cases. — Rami Hassan
– Rami Hassan, Commented Nov 20, 2020 at 16:35
What I mean is you have way too much code in your question and I personally can't follow what it is that you want. — D. SM
– D. SM, Commented Nov 20, 2020 at 16:37
With that, I couldn't agree more, But also I struggled with simplifying the code. I will try to give it one more shot. — Rami Hassan
– Rami Hassan, Commented Nov 20, 2020 at 21:48
What I'm trying to achieve is to check if the Python C extension module is installed, if yes continue if not install it reload modules then continue. — Rami Hassan
– Rami Hassan, Commented Nov 20, 2020 at 23:18

Rami Hassan · Accepted Answer · 2020-11-22 14:39:34Z

Finally, I found the solution by trial and error. for this example code id did the following:

from utils.my_custom_py import MyCustomExtended
from importlib.util import find_spec
from importlib import reload
from sys import modules
from os import system


def setup():
    if system('./setup.py clean build install') > 0:
        raise SystemError("Failed to setup python c-api extention class")

def reload_my_libs():
    global MyCustomExtended
    reload(modules['utils'])
    reload(modules['utils.my_custom_py'])
    from utils.my_custom_py import MyCustomExtended


def main():
    if find_spec('custom2') is None:
        setup()
        reload_my_libs()

    MyCustomExtended(1, 2, 3)
    print("COOL")


if __name__ == "__main__":
    main()

as a result, I got:

running clean
running build
running build_ext
building 'custom2' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/src
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/tmp/PlayAround/.venv/include -I/usr/include/python3.6m -c src/custom.c -o build/temp.linux-x86_64-3.6/src/custom.o
creating build/lib.linux-x86_64-3.6
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/src/custom.o -o build/lib.linux-x86_64-3.6/custom2.cpython-36m-x86_64-linux-gnu.so
running install
running install_lib
copying build/lib.linux-x86_64-3.6/custom2.cpython-36m-x86_64-linux-gnu.so -> /tmp/PlayAround/.venv/lib/python3.6/site-packages
running install_egg_info
Removing /tmp/PlayAround/.venv/lib/python3.6/site-packages/custom2-1.0.egg-info
Writing /tmp/PlayAround/.venv/lib/python3.6/site-packages/custom2-1.0.egg-info
COOL

It also worked for my previous question version, which was way more complicated.

regardless, thanks for the effort to help :)

I would add for whoever encounters such an issue in the future, in case you have a more complicated project than this example, as this question previous version or more, the reloading order matters.

D. SM · Accepted Answer · 2020-11-22 03:20:45Z

1

Besides the various minor issues in your code, you are also trying to reload all the modules in existence while silently swallowing errors. Reloading the utils module works as expected:

import utils
from importlib.util import find_spec
from importlib import reload
from os import system, stat
import weakref
import sys


def setup():
    if system('python3 setup.py clean build install') > 0:
        raise SystemError("Failed to setup python c-api extention class")


def main():
    if find_spec('custom2') is None:
        setup()
        for module_name in list(sys.modules.keys()):
            m = sys.modules.get(module_name)
            if not hasattr(m, '__file__'):
                continue
            if getattr(m, '__name__', None) in [None, '__mp_main__', '__main__']:
                continue

        reload(utils)

    MyCustomExtended = utils.MyCustomExtended
    print(MyCustomExtended)
    MyCustomExtended(1, 2, 3)
    print("COOL")


if __name__ == "__main__":
    main()

I suggest you only reload modules that are part of your application.

answered Nov 22, 2020 at 3:20

D. SM

14.6k3 gold badges15 silver badges22 bronze badges

3 Comments

Rami Hassan Over a year ago

one thing that I know for sure, in order to solve this, case it's solvable, I need to understand what import exactly do.

D. SM Over a year ago

The results I got match your "working" example, update your question to clarify.

Rami Hassan Over a year ago

I updated (or oversimplified to be more exact the question) because you said and I quote: “you have way too much code in your question and I personally can't follow what it is that you want“, the main issue was in my first question is the order of the reloads

Collectives™ on Stack Overflow

Python3 reload project that use python c-api

2 Answers 2

1 Comment

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related