How to include a directory with compiled binaries in a Python wheel using pyproject.toml?

Question

I want to create a Python wheel that contains a single Python module (script.py) and a directory of compiled C++ binaries. Below is my project structure:

root/
│
├── bin/
│   ├── binary1
│   ├── binary2
│   ├── binary3
│   └── binary4
├── script.py
├── pyproject.toml
├── MANIFEST.in
├── README.md

pyproject.toml:

[build-system]
requires = ["setuptools>=61.0.0", "wheel"]
build-backend = "setuptools.build_meta"

[tool.setuptools]
py-modules = ["script"]

[project]
name = "script-test"
description = "Lorem ipsum"
readme = "README.md"
license = { file = "LICENSE" }
authors = [
  { name = "", email = "[email protected]" },
]
requires-python = ">=3.7"
dynamic = ["version"]
classifiers = [
  "Programming Language :: Python :: 3",
  "License :: OSI Approved :: GNU Affero General Public License v3",
  "Natural Language :: English",
  "Intended Audience :: Science/Research",
  "Topic :: Scientific/Engineering :: Bio-Informatics",
  "Operating System :: MacOS",
  "Operating System :: POSIX :: Linux"
]

[tool.setuptools.dynamic]
version = {attr = "script.__version__"}

[tool.setuptools.package-data]
"*" = ["bin/*"]

[project.scripts]
script = "script:main"

MANIFEST.in

include README.md
include LICENSE
recursive-include bin *

After running:

python -m build

The wheel (script_test-0.0.4-py3-none-any.whl) is created, but the bin/ directory is not included in the wheel, while it is correctly included in the source distribution (script_test-0.0.4.tar.gz).

Build output (truncated):

* Building wheel...
/private/var/folders/1k/dnrtvbx9445dpk1shw82v9fw0000gn/T/build-env-jgwgjjff/lib/python3.12/site-packages/setuptools/config/expand.py:126: SetuptoolsWarning: File '/private/var/folders/1k/dnrtvbx9445dpk1shw82v9fw0000gn/T/build-via-sdist-l75sfsez/script_test-0.0.4/LICENSE' cannot be found
  for path in _filter_existing_files(_filepaths)
running bdist_wheel
running build
running build_py
creating build/lib
copying script.py -> build/lib
running egg_info
writing script_test.egg-info/PKG-INFO
writing dependency_links to script_test.egg-info/dependency_links.txt
writing entry points to script_test.egg-info/entry_points.txt
writing top-level names to script_test.egg-info/top_level.txt
reading manifest file 'script_test.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'LICENSE'
writing manifest file 'script_test.egg-info/SOURCES.txt'
installing to build/bdist.macosx-10.13-universal2/wheel
running install
running install_lib
creating build/bdist.macosx-10.13-universal2/wheel
copying build/lib/script.py -> build/bdist.macosx-10.13-universal2/wheel/.
running install_egg_info
Copying script_test.egg-info to build/bdist.macosx-10.13-universal2/wheel/./script_test-0.0.4-py3.12.egg-info
running install_scripts
creating build/bdist.macosx-10.13-universal2/wheel/script_test-0.0.4.dist-info/WHEEL
creating '/Users/andrzej/Desktop/vclust_env/vclust_test/dist/.tmp-8zpx71kj/script_test-0.0.4-py3-none-any.whl' and adding 'build/bdist.macosx-10.13-universal2/wheel' to it
adding 'script.py'
adding 'script_test-0.0.4.dist-info/METADATA'
adding 'script_test-0.0.4.dist-info/WHEEL'
adding 'script_test-0.0.4.dist-info/entry_points.txt'
adding 'script_test-0.0.4.dist-info/top_level.txt'
adding 'script_test-0.0.4.dist-info/RECORD'
removing build/bdist.macosx-10.13-universal2/wheel
Successfully built script_test-0.0.4.tar.gz and script_test-0.0.4-py3-none-any.whl

I would like the bin/ directory to be part of the wheel, at the same level as script.py, and for the binaries to be installed into the Python site-packages/ directory alongside the script. How can I include the bin/ directory in the wheel and ensure it is installed with the package?

You should not do this unless you can guarantee that your binaries run on macOS, Linux, Windows, all implementations etc. Because that is what the filename compatibility tag -py3-none-any.whl is indicating, and these flags indicating the binary compatibility is the whole point of the wheel packaging format in the first place. — wim
– wim, Commented Oct 13, 2024 at 15:38
Instead of packaging executables as "package data", look at a project such as cibuildwheel to build the executables from source and tag the wheels appropriately. — wim
– wim, Commented Oct 13, 2024 at 15:39
I do have separate binaries for macOS, Linux, and Windows. I would like to know why the bin directory is not included in the wheel. — sherlock85
– sherlock85, Commented Oct 13, 2024 at 15:51
Besides what wim said... for data to be packaged in the wheel and installed, the data files have to be part of the import package. Since you do not have any import packge, you can not have package data. -- Recommendation is to create a script directory, move the content of script.py to script/__init__.py, and then move the bin directory to script/bin. Finally you will have to find the correct setuptools settings: setuptools.pypa.io/en/latest/userguide/datafiles.html — sinoroc
– sinoroc, Commented Oct 13, 2024 at 16:24

wim · Accepted Answer · 2024-10-13 16:25:50Z

2

The issue is that you don't have a package at all, you have a single module distribution root/script.py:

py-modules = ["script"]

To package data files, you'll need to have a package like root/mypackage/__init__.py and use:

packages = ["mypackage"]

If you restructure from single-module distribution into a package, you may want to consider to refactor the script.py into a console-script entrypoint within the package (ref).

answered Oct 13, 2024 at 16:25

wim

368k112 gold badges681 silver badges816 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to include a directory with compiled binaries in a Python wheel using pyproject.toml?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related