20

Note

I realise this could be subjective question to answer, but I don't expect an answer in the form of "Do this and everything will work out". I am aware there will probably be some decisions to be taken and that different approaches are better for different scenarios. In any case, I would like a review of the pros and cons of whatever strategies professionals are actually using.

Assume that I am writing a big library, called BigLibrary. This library can be divided into several smaller projects, some of which are also useful outside of BigLibrary. What is the best way to organise the folder structure and installation to have a smooth workflow?

Consider that I also use git and that multiple developers should be able to work on all aspects of the project.

I have some options:

Option 1

BigProject 
  __init__.py
  main_files.py
  subLibrary1
  subLibrary2

The problem with this is that subLibrary1 and subLibrary2 must be accessed through BigProject, which is not ideal. Also it complicates the workflow with git if I want subLibrary1 and subLibrary2 to have their own git repository.

Option 2

BigProject
  __init__.py
Library1
Library2

The problem with this is that BigProject will not see Library1 and Library2, unless I add them manually to sys.path. This seems like an ugly solution. It does make everything work nicely with git, but it also pollutes the namespace a bit.

Ideally, I should also be able to modify Library1 and Library2 without this affecting BigProject - and only deciding to update to a certain version of Library1 and Library2 when I feel like it is a good moment. Basically I want to make BigProject use a certain checkout of Library1 and 2 until I decide to "update it" and bring it to the current checkout.

Improve this question
2
  • 1
    Have you looked at submodules? Commented Feb 8, 2018 at 11:26
  • @Mael Very good resource, it seems to fit my needs. Thanks a lot! Commented Feb 8, 2018 at 14:19

1 Answer 1

Reset to default
22

Neither. Make it possible to install the libraries via pip, then pin a specific version through a git submodule. In detail:

Don't separate your sub-libraries through a particular directory layout. And don't mess around with sys.path. Instead, make your libraries separately installable. This likely means a separate git repository, complete with their own setup.py. This is confusing the first time you do it, but after that the setup.py mostly stays the same. Some configuration options are only relevant when distributing the library through PyPI, that is not the case for you.

You can now check out the library, and install it like pip install path/to/library. But how do we specify which version of the dependency should be installed?

The solution is to pin a specific version through a git submodule. This includes the library repository as a folder in your main repository, at a specific version. You could put all your libraries in a common folder, e.g. lib/library1 and lib/library2. Git submodules take a bit to get used to, because you need to create a commit in the main repository when you want use a new version of the submodule repository – the submodule is viewed as a single file that contains the submodule's current commit hash.

You can then install the libraries as editable modules, e.g. pip install -e lib/library1. The --editable mode means that changes in the source code of the libraries are directly available when importing that library. Other changes may require you to reinstall the library (remember to add the --upgrade flag when reinstalling).

It is sensible to create a virtualenv for your main project. This is not committed to the repository, but avoids polluting the global Python installation.

One drawback of managing dependencies locally is that you cannot use pip's built-in dependency resolution. If your scenario becomes more complicated, it might be helpful to have an internal PyPI mirror to which your dependencies are uploaded. You can then instruct pip to satisfy dependencies from there. But that is likely overkill when getting started.

Suggested directory layout:

bigProject/
  lib/
    library1/
      library1/
        __init__.py
      setup.py
    library2/
      library2/
        __init__.py
      setup.py
  bigProject/
    __init__.py
  requirements.txt

The requirements.txt file can feed instructions to pip, and would contain the paths to the submodules.

Inside your code, you can then use the installed libraries just like normal libraries:

import library1
Improve this answer
4
  • 1
    Thanks a lot! This is exactly the kind of answer I was looking for. I do have one question though, which is: why do I need to install it with pip? I have library1 and library2 living alongside BigProjects, all of them with separate repositories. Then I add library1 and library2 as submodules into BigProject/lib; this means that I can checkout any version I want and it would work fine, I can change library1 without impacting BigProject (unless I pull the changes). So far so good, this is all that I require. Why do I need to do a pip installation then? Commented Feb 8, 2018 at 13:41
  • Also, I want the BigProject to be essentially self contained. So when someone clones it (and I guess installs it), then you can do BigProject.library1 and BigProject.library2, without having library1 and library2 as global python imports Commented Feb 8, 2018 at 14:16
  • 1
    @Ant I strongly suggest installing the libraries via pip because that doesn't constrain the future evolution of these libraries. If the library repos were to have their __init__.py directly at the top level, that can never be changed. In contrast, a separate setup.py for each library makes it possible to organize and test each library independently. You can still make these libraries be accessible as bigproject.library1 by using namespace packages, a slightly advanced topic. A pip install is the easiest and least fragile way to permanently put the libraries in the python-path. Commented Feb 8, 2018 at 15:41
  • 1
    Just to add: if one doesn't like git submodules then there's also git subtree. Commented Feb 8, 2018 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.