pyOpenSci website infrastructure#

pyOpenSci uses GitHub to manage almost all of its infrastructure, from community processes to website rendering. This page gives a high-level overview of how our repositories work together to build our websites and keep community data up to date.

Note

For details on each part of our infrastructure, see the Learn more section below.

How our websites fit together#

The pyOpenSci website at pyopensci.org is built from six repositories, in three groups. Each repository is built separately and published under the pyopensci.org domain using GitHub Pages. Netlify builds a preview of each pull request so you can see changes before they are merged.

The six repositories are:

Core website: pyopensci.github.io, built with Hugo. It is the central hub that links to every sub-site.
Sphinx books: the Python Package Guide, Software Peer Review Guide, Handbook, and Lessons. Most use our branded pyos-sphinx-theme. That theme is built on pydata-sphinx-theme, the Sphinx theme maintained by the PyData community. We add pyOpenSci colors, type, and layout on top of it, so the books share one look and a branding change happens in one repository.
Metrics dashboards: metrics, built with Quarto.

../../_images/website-repositories-structure.svg — pyOpenSci website structure diagram showing the main website and its sub-sites (Handbook, Python Package Guide, Software Peer Review Guide, Lessons, and Metrics).#

How data flows from our community programs to the website#

Some parts of our websites update automatically using data from GitHub. This includes the peer review numbers in our metrics dashboards, and the package listings, community contributors, and editorial board on the main Hugo website.

We wrote a Python package called pyosMeta to collect this data. It saves the data as YAML files, a plain-text format that our websites can read. pyosMeta collects three kinds of data:

Peer review data. pyosMeta parses the peer review issues in software-submission. It pulls out metadata such as the package name, repository URL, editor, reviewers, maintainers, and authors. Maintainers and authors are different people. That data feeds the Python packages page.
Peer review roles. We track editors, the Editor in Chief, and the triage team using GitHub teams. pyosMeta reads team membership to build the editorial board listing. See editorial teams.
Contributors. We use the All Contributors bot to track contributors across our repositories. pyosMeta parses each repository’s All Contributors file to build the contributor list on the Our Community page.

The diagram below shows how this data moves from GitHub to the website and metrics dashboards:

        flowchart TD
  Contrib[All Contributors files] --> Meta[pyosMeta]
  Issues[Peer review issues in software-submission] --> Meta
  Teams[GitHub editorial teams] --> Meta
  Meta --> PR[Pull request with updated YAML files]
  PR -->|merged| Website[Website pages]
  Website -.->|editorial dashboard reads board data| Metrics[Metrics dashboards]
  Issues -.->|review data| Metrics

Scheduled workflows run pyosMeta and open a pull request with the updated YAML files. The website updates once that pull request is merged. The metrics dashboards run on their own schedule. A few fields, including advisory council membership and leadership roles, are edited by hand. Schedules, how to run a workflow, and the full list of hand-edited fields are on website and metrics data.

Learn more#

For details on each part of our infrastructure, see:

All repositories: Complete list and description of all pyOpenSci GitHub repositories
Website & metrics data: Schedules, files, and troubleshooting for contributor, package, and editorial data
Editorial teams: Which GitHub team to use when you add or remove an editor
Permissions: Repository access management and team structures

To contribute to our repositories, see:

Issues: Issue management and labeling workflows
Pull requests: How to work with pull requests in pyOpenSci repos
Continuous integration: The automated checks that run on every pull request, and what to do when one fails