Including a data folder in a python package using setup.cfg

Question

I have the following setup.cfg file, a usual project I want to publish to pypi and which should be shipped with a data folder, in my case ru_core_news_sm-3.1.0. (I am using setuptools):

[metadata]
name = test-project-name
version = 0.6.0

[options]
packages = find:
python_requires = >=3.6
include_package_data = True
install_requires = 
    spacy==3.1.3
    beautifulsoup4

[options.package_data]
test_project_name/ru_core_news_sm-3.1.0 = *.*

This is my directory structure

base_project_name/
├── test_project_name
│   ├── __init__.py
│   ├── a.py
│   └── ru_core_news_sm-3.1.0
├── pyproject.toml
├── MANIFEST.in
└── setup.cfg

and I have this in my MANIFEST.in:

recursive-include test_project_name/ru_core_news_sm-3.1.0 *.*

I am testing the install by executing pip install path/to/base_project_name

However, this does not get me all files in the data folder, but only some of them, and I don't know why.

These are the files in the base folder:

and this is what remains when I have "test installed" it using pip to some other location (both projects use venv):

What could be the reason?

Ewout ter Hoeven · Accepted Answer · 2022-09-03 15:46:32Z

3

Starting with Setuptools 62.3.0, you can now use recursive wildcards ("**") to include a (sub)directory recursively. This way you can include whole folders with all their folders and files in it.

For example, when using a pyproject.toml file, this is how you include two folders recursively:

[tool.setuptools.package-data]
"ema_workbench.examples.data" = ["**"]
"ema_workbench.examples.models" = ["**"]

But you can also only include certain file-types, in a folder and all subfolders. If you want to include all markdown (.md) files for example:

[tool.setuptools.package-data]
"ema_workbench.examples.data" = ["**/*.md"]

It should also work when using setup.py or setup.cfg.

See https://github.com/pypa/setuptools/pull/3309 for the details.

answered Sep 3, 2022 at 15:46

Ewout ter Hoeven

4115 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Eric Duminil Over a year ago

Excellent, thanks. I've used your config as a starting point. I've also converted my setup.cfg to pyproject.toml with ini2toml setup.cfg -o pyproject.toml. I now have dynamic version, and included data files in my wheels. And I finally removed setup.py and setup.cfg, which caused me many headaches.

Pux · Accepted Answer · 2022-04-29 15:53:52Z

2

I found the problem: You should not use *.* to include all files, but instead simply *. Because sometimes files don't have an ending, which will then get ignored. Depending on the file (MANIFEST.in):

recursive-include project_name/ru_core_news_sm-3.1.0 *

or (setup.cfg)

test_project_name/ru_core_news_sm-3.1.0 = *

answered Apr 29, 2022 at 15:53

Pux

5871 gold badge6 silver badges24 bronze badges

Collectives™ on Stack Overflow

Including a data folder in a python package using setup.cfg

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related