Update website and metrics data#

Note

This page is for editors, staff, and repository maintainers who keep our website and dashboards up to date. You don’t need it to contribute to pyOpenSci. To learn how our websites fit together, see the infrastructure overview.

This page explains how community, package, and editorial data get updated on the pyOpenSci website and our metrics dashboards, and how you can update the data yourself when needed. We use a Python package called pyosMeta to collect data about our peer review process and community contributors. pyosMeta saves this data as YAML files, a plain-text format that our website and metrics dashboards can read. GitHub Actions workflows run pyosMeta every week on a schedule (a cron job). Each run opens a pull request with the updated data. You can also run a workflow by hand whenever you need to. The workflows live in two repositories:

Website: the workflows are in the pyopensci.github.io repository. They update the pages on pyopensci.org.
Metrics dashboards: the workflows are in the metrics repository. They update the dashboards at pyopensci.org/metrics.

Once that pull request is merged, the website or dashboards update.

The infrastructure team usually merges these pull requests. If something isn’t updating, post in the #pyos-maintainers-infrastructure channel in Slack.

Where the data comes from#

pyosMeta collects three kinds of data from GitHub:

Peer review data. pyosMeta parses the peer review issues in software-submission. It pulls out metadata such as the package name, repository URL, editor, reviewers, maintainers, and authors. Maintainers and authors are different people. That data feeds the Python packages page.
Peer review roles. We track editors, the Editor in Chief, and the triage team using GitHub teams. pyosMeta reads team membership to build the editorial board listing, including who is current and who is emeritus. See editorial teams.
Contributors. We use the All Contributors bot to track contributors across our repositories. pyosMeta parses each repository’s All Contributors file to build the contributor list on the Our Community page.

        flowchart TD
  Contrib[All Contributors files] --> Meta[pyosMeta]
  Issues[Peer review issues in software-submission] --> Meta
  Teams[GitHub editorial teams] --> Meta
  Meta --> PR[Pull request with updated YAML files]
  PR -->|merged| Website[Website pages]
  Website -.->|editorial dashboard reads board data| Metrics[Metrics dashboards]
  Issues -.->|review data| Metrics

Which workflow updates which page#

Website workflows#

These workflows are in the pyopensci.github.io repository.

Our Community
- File: data/contributors.yml
- Workflow: Update Contribs & reviewers
- Runs: Mondays, 03:21 UTC
Python packages
- File: data/packages.yml
- Workflow: Update Contribs & reviewers
- Runs: Mondays, 03:21 UTC
Editorial board
- Files: data/editorial-board.yml, data/emeritus-editors.yml, and the editorial fields in data/contributors.yml
- Workflow: Update editorial board
- Runs: Wednesdays at 04:21 UTC, or any time you run it by hand. The Monday workflow refreshes these files too.

Metrics workflows#

These workflows are in the metrics repository.

Metrics dashboards
- Files: CSVs in the data/ directory of the metrics repository
- Workflow: Update issue, pr and contrib metadata
- Runs: 05:00 UTC on the 2nd, the 16th, and every Monday, plus December 31

Which workflow should I run?#

Changed an editorial team member? Run Update editorial board in pyopensci.github.io. It only looks at team membership, so it runs quickly and opens a small pull request that is easy to review.
Need contributor or package updates? Run Update Contribs & reviewers in pyopensci.github.io. It checks every repository and review issue, so it takes longer and opens a much larger pull request.
Need fresh metrics data? Run Update issue, pr and contrib metadata in metrics.

Note

The editorial charts in the metrics dashboards read the website’s YAML files from the data/ directory on the main branch. A separate deploy workflow in the metrics repository rebuilds the dashboards every Sunday at 00:00 UTC using whatever has been merged by then, so you may see a short delay.

Run a workflow now#

You don’t have to wait for the next scheduled run. If you have write access to the repository, you can run any of its workflows yourself.

Update the website#

The website workflows are in pyopensci.github.io. The Editor in Chief team, the Software Review Lead, and the repository maintainers team have write access there.

Open the Actions tab in pyopensci.github.io.
Choose the workflow you need. If you changed an editorial team, pick Update editorial board. For contributor or package updates, pick Update Contribs & reviewers.
Select Run workflow.
When the pull request opens, take a quick look and merge it if everything looks right.

Update the metrics dashboards#

The metrics workflows are in the metrics repository.

Open the Actions tab in metrics.
Choose Update issue, pr and contrib metadata.
Select Run workflow.
When the pull request opens, review it and merge it.

The dashboards show the new data after the next deploy. See the note above about the Sunday rebuild.

Not sure whether you can change a team or run a workflow? The editorial teams page explains who can do what.

Fields you edit by hand#

The workflows overwrite data/packages.yml, data/editorial-board.yml, and data/emeritus-editors.yml, so don’t edit those by hand.

You can edit some fields in data/contributors.yml by hand. Once those edits are on main, the Monday workflow keeps them:

Advisory status (advisory). See Update the advisory council.
Leadership status (board). Set this to true to list someone under Executive Council, Leadership & Staff.
Title (title), the role shown on their card. Editor titles still come from the editorial workflow. Other titles, such as Advisory Council or Executive Director, stay as you write them.
Name (name). A users name on GitHub is sometimes missing and also not always someone’s preferred name, so you can update it here. Our data procedss workflow never overwrites a name that is already in the file.

Editors who can’t join our GitHub organization go in data/manual-editorial-roster.yml. See editorial teams.

Important

Merge your pull request before Monday’s Update Contribs & reviewers run. If that pull request is already open, add your edit to it. Merging an older bot pull request after your edit can drop your change.

Update the advisory council#

The Advisory Council on the Our Community page is maintained by the core team. There is no GitHub team for it, and no workflow to run. Edit data/contributors.yml in the website repository, open a pull request, and merge it. The page updates on the next website deploy.

The website lists current members alphabetically by first name. Names in the file are stored as first name, then last name. You do not set sort for this section.

Add a current member#

Find the person in data/contributors.yml. Before advisory, emeritus_advisory, and title are set, the entry looks like this:

- name: Jane Doe
  github_username: janedoe
  github_image_id: 1234567
  title:
  sort:
  bio:
  organization:
  date_added: '2024-01-15'
  editorial_board: false
  emeritus_editor: false
  advisory: false
  emeritus_advisory: false
  twitter:
  mastodon:
  orcidid:
  partners:
  website:
  board: false
  contributor_type:
    - community
  packages_eic:
  packages_editor:
  packages_submitted:
  packages_reviewed:
  location:
  email:

If they are not in the file yet, add an entry for them with the same fields as the other entries. Then set advisory, emeritus_advisory, and title:

- name: Jane Doe
  github_username: janedoe
  github_image_id: 1234567
  title: Advisory Council
  sort:
  bio:
  organization:
  date_added: '2024-01-15'
  editorial_board: false
  emeritus_editor: false
  advisory: true
  emeritus_advisory: false
  twitter:
  mastodon:
  orcidid:
  partners:
  website:
  board: false
  contributor_type:
    - community
  packages_eic:
  packages_editor:
  packages_submitted:
  packages_reviewed:
  location:
  email:

If they already have a title, such as an editor title, make title a list and include Advisory Council as one item.

A new entry also needs:

name, with the first name first
github_username
github_image_id, the numeric GitHub user id used for their photo. Read the id field from https://api.github.com/users/USERNAME

When someone steps down#

Set:

advisory: false
emeritus_advisory: true
title: Emeritus Advisory Council

emeritus_advisory: true places someone in the Emeritus Leadership section. That section also includes other emeritus leadership, such as emeritus Executive Council. Leave the flag in place for anyone who should stay in that section.

If something breaks#

Don’t worry, you won’t break anything by running a workflow. If one fails, a message is posted automatically to the #pyos-maintainers-infrastructure Slack channel. If you get stuck, post there too, and the pyOpenSci repository maintainers will be happy to help. If you’d like to dig in yourself, the pyosMeta development guide explains the tokens and permissions the workflows use.

Review issues that don’t parse#

pyosMeta reads the text people type into the review issue template. People fill out templates in all sorts of creative ways, so a missing field or an extra line can trip it up. If a workflow fails:

Post a link to the logs in #pyos-maintainers-infrastructure. The error may already be there from the Slack automation, but a post from you helps us prioritize the fix.
Check the logs for the package it failed on. Then look at the YAML at the top of that review issue for quirks. For example, text in quotes "like this" can break parsing.