2

I would need to deploy platform-specific package data in the wheels built with setup.py

For a bit of background, I am trying to create Python bindings for a certain library using pre-built binaries and ctypesgen. I have a data tree with platform-specific subdirectories that contain a bindings file and a binary each:

data/
    macos-arm64/
        _bindings.py
        binary.dylib
    linux-x64/
        _bindings.py
        binary.so
    windows-x64/
        _bindings.py
        binary.dll
    ...

And I have a source tree:

src/
    package_name/
        __init__.py
        __main__.py
        ...

Question: How can I build platform-specific packages that only contain the one corresponding binary and bindings?

The final directory tree as deployed to the end-user should structurally look like this:

package_name/
    __init__.py
    __main__.py
    _bindings.py
    binary.xyz
    ...
Improve this question

2 Answers 2

Reset to default
2

This is the solution I am currently using for pypdfium2:

  • Create a class of supported platforms whose values correspond to the data directory names:
class PlatformNames:
    darwin_x64  = "darwin_x64"
    linux_x64   = "linux_x64"
    windows_x64 = "windows_x64"
    # ...
    sourcebuild = "sourcebuild"
  • Wrap setuptools.setup() with a function that takes the platform name as argument and copies platform-dependent files into the source tree as required:
# A list of non-python file names to consider for inclusion in the installation, e. g.
Libnames = (
    "somelib.so",
    "somelib.dll",
    "somelib.dylib",
)

# _clean() removes possible old binaries/bindings
# _copy_bindings() copies the new stuff into the source tree
# _get_bdist() returns a custom `wheel.bdist_wheel` subclass with the `get_tag()` and `finalize_options()` functions overridden so as to tag the wheels according to their target platform.

def mkwheel(pl_name):
    _clean()
    _copy_bindings(pl_name)
    setuptools.setup(
        package_data = {"": Libnames},
        cmdclass = {"bdist_wheel": _get_bdist(pl_name)},
        # ...
    )
    # not cleaning up afterwards so that editable installs work (`pip3 install -e .`)
  • In setup.py, query for a custom environment variable defining the target platform (e. g. $PYP_TARGET_PLATFORM).
    • If set to a value that indicates the need for a source distribution (e. g. sdist), run the raw setuptools.setup() function without copying in any build artifacts.
    • If set to a platform name, build for the requested platform. This makes packaging platform-independent and avoids the need for native hosts to craft the wheels.
    • If not set, detect the host platform using sysconfig.get_platform() and call mkwheel() with the corresponding PlatformNames member.
      • In case the detected platform is not supported, trigger code that performs a source build, moves the created files into data/sourcebuild/ and runs mkwheel(PlatformNames.sourcebuild).
  • Write a script that iterates through the platform names, sets your environment variable and runs python3 -m build --no-isolation --skip-dependency-check --wheel for each. Also invoke build once with --sdist instead of --wheel and the environment variable set to the value for source distribution.

→ If all goes well, the platform-specific wheels and a source distribution will be written into dist/.

Perhaps this is a lot easier to understand just by looking at pypdfium2's code (especially setup.py, setup_base.py and craft_packages.py).

Disclaimer: I am not experienced with the setup infrastructure of Python and merely wrote this code out of personal need. I acknowledge that the approach is a bit "hacky". If there is a possibility to achieve the same goal while using the setuptools API in a more official sort of way, I'd be interested to hear about it.

Update 1: A negative implication of this concept is that the content wrongly ends up in a purelib folder, although it should be platlib as per PEP 427. I'm not sure how to instruct wheel/setuptools differently. Luckily, this is rather just a cosmetic problem.

Update 2: Found a fix to the purelib problem:

class BinaryDistribution (setuptools.Distribution):
    def has_ext_modules(self):
        return True

setuptools.setup(
    # ...
    distclass = BinaryDistribution,
)
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

-1

Not quite what you described, but for simplicity I'd ship a wheel that is built for all supported platforms.

setup.py

import setuptools

setuptools.setup(
    name='package_name',
    packages=[
        'macos-arm64',
        'linux-x64',
        ...
    ],
    include_package_data=True
)

MANIFEST.in

include macos-arm64/binary.dylib
include linux-x64/binary.so
...

You'd need to adjust your import code a bit to import the correct bindings depending on platform.

The setuptools documentation describes in more detail how to do this.

Improve this answer

5 Comments

Packaging all binaries into a single wheel is not a good solution for several reasons. Depending on how big the library is and how many platforms are supported, this can make your package ridiculously large. Moreover, if someone uses a platform for which you don't have binaries, it will lead to strange errors instead of a coherent report by pip that there's no suitable wheel available. Anyway, I already found an acceptable solution how to do get the binaries into platform-specific wheels. If I have some more time I can post an answer about it.
@mara004 could you post your solution?
@vitalstatistix Done. Please see my answer below. I hope this helps you. If you have any questions left, feel free to ask.
@frampy Also note that your approach isn't actionable if aiming to supporting multiple architectures of one OS (e. g. macOS Intel and ARM64) if the binary loader doesn't offer a smart way to choose the right file. For instance, ctypesgen will correctly distinguish between .so/.dll/.dylib for Linux/Windows/macOS, but it doesn't have a way to choose between two binaries for the same OS, so you'd have to inject custom code into the loader by patch, which is tedious and prone to changes in the bindings generator.
MANIFEST.in should only include source distribution (sdist) files, not compiled objects

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.