Structuring a python project

Question

I need to know how to set up the __init__.py and imports in order to structure a python project where I can use fully qualified names throughout the package.

The package will contain a number of sub packages which may contain clashing names. The classes contained within the package will sub class each other and contain references to each other. The project will be generated so the use of fully qualified names would make life a lot simpler.

This sample project represents the structure I am aiming at, but only contains a single sub project, while the IDE seems happy with it fails when its run.

MyPackage/__init__.py

import SubPackage as SubPackage

MyPackage/SubPackage/__init__.py

from .FileB import ClassB
from .FileA import ClassA

MyPackage/SubPackage/FileA.py

from __future__ import absolute_import
import MyPackage


class ClassA(MyPackage.SubPackage.ClassB):
    thingA: 'MyPackage.SubPackage.ClassA'
    thingB: MyPackage.SubPackage.ClassB

    def __init__(self):
        self.thingA = None
        self.thingB = None

    def test(self):
        self.thingA = MyPackage.SubPackage.ClassA()
        self.thingB = MyPackage.SubPackage.ClassB()

MyPackage/SubPackage/FileB.py

from __future__ import absolute_import
import MyPackage


class ClassB(object):
    nextB: 'MyPackage.SubPackage.ClassB'

    def __init__(self):
        self.nextB= None

    def test(self):
        self.nextB= MyPackage.SubPackage.ClassB()

test.py

import MyPackage

x = MyPackage.SubPackage.ClassA()

Error

  File "C:\temp\Test.py", line 3, in <module>
    import GeneratedLx
  File "C:\temp\MyPackage\__init__.py", line 1, in <module>
    import Bs as Bs
  File "C:\temp\MyPackage\SubPackage\__init__.py", line 12, in <module>
    from .FileA import ClassA
  File "C:\temp\MyPackage\SubPackage\FileA.py", line 5, in <module>
    class ClassA(MyPackage.SubPackage.ClassB):
AttributeError: module 'MyPackage' has no attribute 'SubPackage'

Why do you need to fully qualify everything? Why even have so many modules? If the code in the question is representative, your modules are very short. You would usually just include several classes in each module, avoiding the need to import so much. Is there a reason not to? Furthermore, unless there are name clashes, shorter references are preferable. — jpmc26
– jpmc26, Commented Jan 29, 2019 at 16:45
There will be potentially 100s of modules and max of about 5 sub packages. The classes in the sub packages may interrelate (derive from classes in another subpackage) and use classes in other subpackages. So there is the potential for name clashes, so fully qualifying the names would be preferable. — Sprotty
– Sprotty, Commented Jan 29, 2019 at 16:58
Leaving MyPackage off the name chain should leave your variables qualified enough to avoid conflicts, and eliminates the reference problem (chicken and the egg I think). In "FileA" and B, just from MyPackage import Submodule and skip MyPackage. in names. FTR calling these long attribute chains can have a noticible performance impact (attr calls aren't quick). — Aaron
– Aaron, Commented Jan 29, 2019 at 17:11
@Sprotty It really sounds like you would benefit greatly from consolidating your modules into fewer ones. Especially when classes are interdependent on each other, it's much, much simpler to include them in the same module. Modules and packages are not for creating detailed taxonomies. They're for providing large groupings of functions, classes, and variables that work together or for avoiding name conflicts. If you have hundreds of modules inside one package, you're probably doing something non-Pythonic. In your top level package, consider aliases rather than fully qualifying. — jpmc26
– jpmc26, Commented Jan 29, 2019 at 17:35
The source data (that these classes will be generated from) is broken down into a number of 'namespace', so the same name may occur in 2 namespaces. I figured a SubPackage would be a good container for each namespace. If the classes in each namespace were independent then each sub package could be a single module, however its possible to have inter-relations between the classes in each sub package, i.e nsA:cls1 derives from nsB:cls2 which derives from nsA:cls3. A way to guarantee they are defined in the correct order is splitting 1 class/module, unless Python can do forward declarations? — Sprotty
– Sprotty, Commented Jan 29, 2019 at 19:49

Aaron · Accepted Answer · 2019-01-29 17:23:59Z

2

You already cannot have name conflicts at the SubPackage level, so adding MyPackage is entirely redundant, and doesn't work quite the way you're trying to use it. This may be due to when names are bound or something else, but ultimately there should be no instance when you need it. This leaves you to slightly edit the files: "FileA.py", and "FileB.py":

FileA.py

from __future__ import absolute_import
from MyPackage import SubPackage


class ClassA(SubPackage.ClassB):
    thingA: 'SubPackage.ClassA'
    thingB: SubPackage.ClassB

    def __init__(self):
        self.thingA = None
        self.thingB = None

    def test(self):
        self.thingA = SubPackage.ClassA()
        self.thingB = SubPackage.ClassB()

FileB.py

from __future__ import absolute_import
from MyPackage import SubPackage


class ClassB(object):
    nextB: 'SubPackage.ClassB'

    def __init__(self):
        self.nextB= None

    def test(self):
        self.nextB= SubPackage.ClassB()

The import statement is also equivalent to from .. import Subpackage using relative imports rather than absolute if desired. Stylistically I tend to use relative imports to help me quickly pick out which imports are part of my project, and which ones are external dependencies.

answered Jan 29, 2019 at 17:23

Aaron

11.2k1 gold badge27 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Sprotty Over a year ago

Oddly that seems to work, even when a second SubPackage is added. I have to say, I don't really understand the original error, but this looks like a workable solution.

Aaron Over a year ago

@Sprotty I don't think packages are intended to import themselves at a top level.. you have to go at least one level down (ie: subpackage level). The only reason to go up a level would be to import a completely separate package.

Collectives™ on Stack Overflow

Structuring a python project

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related