Complex Python package builds

This guide is focused on packages that are either pure-python or that have a few simple extensions in another language such as C or C++.

In the future, we want to provide resources for packaging workflows that require more complex builds. If you have questions about these types of package, please add a question to our discourse or open an issue about this guide specifically in the GitHub repo for this guide. There are many nuances to building and distributing Python packages that have compiled extensions requiring non-Python dependencies at build time. For an overview and thorough discussion of these nuances, please see this site.

Pure Python Packages vs. packages with extensions in other languages

You can classify Python package complexity into three general categories. These categories can in turn help you select the correct package frontend and backend tools.

1. Pure-python packages: these are packages that only rely on Python to function. Building a pure Python package is simpler. As such, you can chose a tool below that has the features that you want and be done with your decision!

2. Python packages with non-Python extensions: These packages have additional components called extensions written in other languages (such as C or C++). If you have a package with non-Python extensions, then you need to select a build backend tool that allows additional build steps needed to compile your extension code. Further, if you wish to use a frontend tool to support your workflow, you will need to select a tool that supports additional build setups. We suggest that you chose build tool that supports custom build steps like Hatch.

3. Python packages that have extensions written in different languages (e.g. Fortran and C++) or that have non Python dependencies that are difficult to install (e.g. GDAL) These packages often have complex build steps (more complex than a package with just a few C extensions for instance). As such, these packages require tools such as scikit-build or meson-python to build. NOTE: you can use meson-python with PDM.

Mixing frontend and backend projects

It is sometimes necessary or desirable to use a build frontend with an alternative build-backend. This is because some frontends do not have a default backend (build), and this choice is placed on the maintainer. Other backends (hatch) have a preferred backend (hatchling) but allow the maintainer to migrate to another, while some backends (poetry) only work with a single backend (poetry-core). Refer to (#python-package-build-tools) for more information about frontend and backend compatibility.

In this packaging guide we recommend using hatch along with its preferred backend hatchling. While this will be suitable for most packages, an alternate backend may be used with Hatch if needed when creating an extension module. A Python extension module is one that is made up, either in part or entirely, of compiled code. In this case the backend chosen (such as meson-python) must know how to compile the extension language and bind it to Python. hatchling does not know how to do this all on its own and must either make use of plugins or be replaced by a backend that is already capable of building extension modules.

In order to use a different backend you will need to edit your project’s pyproject.toml. If you have a pyproject.toml generated by the hatch command, or from following the packaging tutorial, you may have to make a change like this

 [build-system]
-requires = ["hatchling"]
+requires = ["meson-python"]
-build-backend = "hatchling.build"
+build-backend = "mesonpy"