Including tests in your package distributions#

Throughout this guide we recommend keeping your tests/ directory outside of your package directory (an “out-of-tree” layout). This keeps your wheel small and, combined with the src layout, helps ensure that your tests run against the installed package rather than the files in your working directory.

Some packages, however, benefit from shipping their tests inside the package (“in-tree” tests). This is common in the scientific Python ecosystem: NumPy, SciPy, and scikit-image all do it.

When in-tree tests make sense#

Consider in-tree tests if your package:

  • Contains compiled code (C, C++, Cython, Rust, or Fortran). A wheel built on one platform may import fine but produce wrong results on another. Letting users run the test suite against the wheel they installed is the most reliable way to find out.

  • Is built by downstream packagers such as conda-forge, Debian, or Spack. Downstream packagers building from an sdist can run tests included in that source distribution, even when tests are excluded from wheels. If supporting these builds is the only goal, including tests in the sdist may be sufficient (see below).

  • Has users who need to verify an installation on an HPC cluster or another locked-down environment where they can’t clone the repository.

If your package is pure Python with a small test suite, out-of-tree tests are simpler, and we recommend sticking with them.

Alternative: include tests in the sdist only#

Tests do not have to be installed with a wheel to be available to downstream packagers. A project can keep its tests/ directory outside the package and include those files in its source distribution (sdist) without including them in the wheel. Downstream packagers building from the sdist can then run the tests without increasing the size of wheels distributed to end users. This supports downstream validation using files from the sdist, but users cannot run those tests from an installed wheel alone.

If you use Hatchling, the build backend used in this guide’s tutorials, you don’t need to configure anything. By default, the sdist includes all files not ignored by your version control system, which covers a top-level tests/ directory, while the wheel contains only your package.

With setuptools, add a MANIFEST.in entry to include an out-of-tree tests/ directory in the sdist:

graft tests

This only affects the sdist; a tests/ directory that sits next to your package is not included in the wheel. See the setuptools guidance on wheel and sdist contents for more detail.

If users need to run tests against an installed wheel, including tests in the importable package as described below may be more appropriate.

Layout#

Move tests/ inside your package directory and give it an __init__.py so that it is discovered and installed like any other subpackage:

src/
  mypackage/
    __init__.py
    core.py
    tests/
      __init__.py
      test_core.py
docs/
pyproject.toml

Make sure the tests ship

Check your build backend’s package-discovery settings. If you list packages explicitly, add mypackage.tests. After building, inspect the wheel (unzip -l dist/*.whl) to confirm that the test files are included.

Run tests against the installed package#

With tests inside the package, run them using pytest’s --pyargs flag. This tells pytest to import mypackage from the active environment, locate its tests subpackage, and run it, regardless of your current working directory:

pytest --pyargs mypackage

Many scientific packages also expose a convenience function so that users don’t need to remember the flag:

# src/mypackage/__init__.py
def test(*args):
    """Run the mypackage test suite."""
    import pytest

    return pytest.main(["--pyargs", "mypackage", *args])
>>> import mypackage
>>> mypackage.test()

Because pytest is imported only when the function is called, it is not required for ordinary package use. If the distributed tests import pytest or the public test() helper shown above uses pytest, declare it as an optional dependency so users can install it when they need to run tests:

[project.optional-dependencies]
test = ["pytest"]

If distributed tests use unittest and do not require pytest, there is no need to expose pytest as an optional dependency solely because developers use it as a test runner. Instead, projects can declare pytest in a development dependency group or manage it through their development environment.

For packages that ship unittest.TestCase tests inside the importable package, users can run an installed test module with Python’s built-in unittest runner without installing pytest:

python -m unittest mypackage.tests.test_core

Replace mypackage.tests.test_core with the fully qualified name of an installed test module. To discover all compatible tests in the installed tests package, use:

python -m unittest discover -s mypackage.tests

The unittest runner does not collect standalone pytest-style test functions and does not provide pytest fixtures. Tests that depend on pytest-specific features may require pytest to run successfully.

If distributed tests or a public test helper require additional third-party dependencies, projects may declare those dependencies in the same test extras group to support running the tests after installation.

Pitfalls#

Running tests on the source tree by accident. This is the main reason we recommend out-of-tree tests for beginners. If you run pytest src/mypackage/tests from the repository root, Python may import your un-built source files instead of the installed package. With compiled code, this surfaces as confusing ImportErrors about missing extension modules. Always use pytest --pyargs mypackage (or a nox or tox session that installs the package first) rather than pointing pytest at a path.

Test data. Don’t ship large datasets inside your wheel. See don’t include test suite datasets in your package for alternatives such as Pooch.

Wheel size. Every test file is downloaded by every user. Keep the in-tree suite focused. Very slow or data-heavy tests can stay out of tree and run only in CI.

For a real-world example, see how NumPy organizes numpy/tests and numpy/<submodule>/tests, and exposes numpy.test().