Partnerships with Scientific Python Communities

Numerous scientific Python communities are developing their own peer review processes for affiliated packages. Communities with their own package review processes can leverage our existing peer review process, which is built on community-driven package standards. In partnership with these communities, we enhance our standard review process with a community-specific step that adds your specific requirements for package affiliation.

Through this single end-to-end review process, a package can potentially:

• Be accepted by pyOpenSci.

• Undergo a fast-track review and get published with JOSS (if in scope).

• Achieve affiliation with your domain-specific community.

Here we discuss our process for supporting this community affiliation partnership.

Scientific communities can leverage our peer review process by adding a layer of review associated with requirements each community has for packages to be labeled as “affiliated” with that specific community. To see an example of such requirements, look at the Pangeo partner page in this guide.

What do we get when we become a pyOpenSci partner?

In addition to the core benefits that pyOpenSci peer review provides, we will provide you with the following:

Visibility for your community and associated Python packages

1. We will create a community page in this peer review guide where you can define your guidelines for a package to become affiliated with your community. Check out the pangeo example as an example of what this looks like.

2. A dedicated community page on the pyOpenSci website. Here, we will list your community along with all affiliated packages that have successfully undergone our review process. You can add descriptive text and graphics and branding to the page as you wish.

Packaging Support

Our vibrant community is filled with experts in the Python packaging space. A partnership with us means that you will always have support if you run into Python packaging issues.

Future work

We are currently working on several items to improve our review process and followup activities with package maintainers including:.

• Collecting metrics and will be creating small dashboards that track package health for all accepted packages in our ecosystem.

• Working on a bot adapted from the JOSS bot that will streamline the editorial process for peer review.

Partnership benefits to the broader scientific community

The benefits for the broader scientific Python community include:

• We have broad reach across the entire scientific Python ecosystem which allows us to promote your community and associated packages.

• We have a broad view across all domains within the scientific Python ecosystem which will help us to identify overlap of package functionality across domains and potentially aggregate efforts in the future as it makes sense (i.e., if a maintainer needs to step down).

• We offer an established peer review process will help enforce standardization of scientific Python packaging and best practices across the ecosystem. Our volunteer-lead peer review process has been operating since 2019.

How peer review works for partners

The pyOpenSci peer review process can be broken down into 5 steps when a package is submitted for review:

1. To kick things off, an Editor-in-Chief checks that the package passes our bare-minimum checks.

2. Once Editor-in-Chief checks are passed, they assign a domain-expert editor to the review.

Through our partnership, we have two options when selecting an editor.

• If the editor is someone from your partner community who has authority, they can determine whether the package meets the decided-upon criteria to become affiliated with your community.

• If we have a domain specific editor who is not authorized by your community to make this decision, we can ask someone from your community to make the decision around affiliated status. Once the decision is made, the review can proceed as normal.

3. Once affiliated status has been determined, the editor finds 2 reviewers to begin the review.

4. The review begins and is a conversation between the package maintainer and the reviewer with the goal of improving the package usability and quality.

5. Once the package is accepted by pyOpenSci, authors can opt to be published in the Journal of Open Source Software (JOSS) if their package is in scope for JOSS. JOSS will accept the pyOpenSci review and will only review the JOSS paper. If the package is within JOSS’s scope, the JOSS fast-track process is normally a quick final step.

Through the above steps, an accepted package can become pyOpenSci accepted, JOSS published and an official affiliated package with your community.

How we find editors and reviewers

One of the most common concerns that communities have is finding editors and reviewers. While finding volunteer editor and reviewers takes time, we are confident that we can figure this out together.

For every review we like to have an editor and at least 1 reviewer with some scientific domain expertise associated with the submitted package (and your community). Where possible we will make calls within your community to find editors and reviewers who understand your domain. However, we also have a broad pool of editors and reviewers across scientific domains that will allow us to fill gaps as needed. For instance, if your community is from the bio-medical single-cell domain we may be able to find editors in the broader field of genomics that could easily manage leading a review as an editor.

We have been running software peer review since 2019 and have never not been able to review a package because of editor / reviewer availability. However, sometimes it might take a bit longer to find the right people to begin the review.

How can people sign up to review?

pyOpenSci maintains a reviewer database with individuals who have signed up using our reviewer sign-up form. Our goal is to assign two reviewers for each package. Typically, one reviewer possesses domain-specific expertise while the second may focus on usability, installation, documentation, or packaging infrastructure, without necessarily needing domain expertise.

We prioritize diversity among contributors in every review, ensuring inclusion of individuals with different gender, cultural backgrounds, and identities.

Frequently asked questions

Below is a list of questions that we get asked about our community partnerships. If you have a question that is now answered here, feel free to ask it on the pyOpenSci discourse forum.

General peer review questions

How do I get my community involved?

We are always open to new partners. Please feel free to ask questions or to leave a post on our discourse forum. We can connect with you from there. You can also email us directly at admin at pyopensci.org.

What if there is a conflict of interest

pyOpenSci’s conflict of interest policy can be found here.

How do I know that a review is community-affiliated?

We will add your community name as a label to our review repository here. This will allow anyone to filter for reviews specific to your community. Here is an example of the community tag for Pangeo.

Affiliated review / pyOpenSci integration questions

Where do affiliated guidelines get published?

The criteria for a package becoming affiliated with a community needs to be clearly communicated on the partner landing page in our peer review guide. These are the criteria that will be evaluated during our review process to determine whether a package should gain affiliated status with your community.

Can a package be accepted by pyOpenSci but not affiliated?

Sometimes, a package may be accepted by pyOpenSci but not meet community affiliation criteria. In such cases, the package will still be listed as an accepted pyOpenSci package. However, it will not receive community affiliation status.

While rare, if a package is rejected by pyOpenSci but the community editor believes it should be affiliated, they can make that call.

In such cases, pyOpenSci won’t list the package on its website, but the community can list it on their page.

When does the package get evaluated for affiliation in your peer review process?

We request that your community appoint a minimum of two individuals to serve on our editorial board. During the initial Editor-in-Chief phase of our peer review process, we assess whether a package aligns with the scope of pyOpenSci. For potential community-affiliated packages, the editor, who is a member of your community, will evaluate the package at the beginning of the review process. We can also perform a second review at the end of the process if requests were made for changes to the package that would allow it to become affiliated with your community.

How do we list pyOpenSci-accepted packages on our website post-review?

You can also access are most recent list of accepted packages using this .yml file which drives our website here. This file is updated using a cron job several times a month and can be filtered by your specific community to support your website development workflow. The full listing of packages can be found here.

Package maintenance

“Rereview” of packages & package maintenance

A core goal of pyOpenSci is to support long-term maintenance of scientific Python packages. To achieve this, we monitor repository activity for all our packages to detect any signs of becoming unmaintained.

However, pyOpenSci does not conduct full re-reviews, involving the Editor-in- Chief, Editor, and two reviewers, for packages that have already been accepted.

Note

We intend to establish automated checks to monitor the “health” and maintenance status of packages over time. These checks aim to identify unmaintained packages. View our process for this here.

Once we identify a tool requiring additional maintenance, we will flag it for our user community.

In the event that a package becomes unmaintained, we will adhere to the current pyOpenSci policies regarding package maintenance post-review.

If it’s a widely-used package and the maintainer wishes to see it continue, we will make efforts to assist the maintainer in finding a new team of maintainers.

Otherwise, we will gracefully sunset it, removing it from our list of actively maintained and accepted packages.

How will you track package maintenance?

We will track package maintenance by collecting metrics such as the commit frequency, releases, and passing tests. Some, if not most, of this data collection will be automated using the GitHub API and other community tools.

Packaging guidelines

How do you develop your packaging guidelines?

Packaging Guidelines

pyOpenSci is developing a community-driven packaging guide that encompasses modern best practices and recommendations for scientific Python packaging. This guide undergoes a stringent community review process involving reviewers from various groups, including members of PyPA, the core Python team, Anaconda, conda/conda-forge maintainers, and maintainers of core packages and front-end/back-end tools (e.g., flit, PDM, hatch/hatchling, etc.). Additionally, it is actively maintained by members of both the scientific Python community and maintainers.

What if we already have community packaging guidelines?

While it’s perfectly fine if your community has its own packaging guidelines, we encourage the broader scientific community to adopt similar packaging best practices. Doing so will enhance the experience for new contributors across different domains. If you have the bandwidth and are interested in collaboration, we welcome contributions to our community-driven pyOpenSci packaging guide.

For communities with existing peer review processes

For communities with existing peer review processes and accepted packages, we recommend the following steps:

1. If you’re in the process of transitioning your review process, start by looking for packages that have recently entered the review phase. Reach out to the maintainers of these packages and see if they’d be interested in shifting their review to pyOpenSci. If you already have an editor leading the review, they can become a guest editor on our board and continue the review using our process. We’re also available to mentor any new editors through our review process.

2. For packages that have already undergone review, consider prioritizing the older ones. Older packages are more likely to become unmaintained over time. This presents an excellent opportunity to re-evaluate packages that were accepted in previous years by taking them through our pyOpenSci peer review process.

Review is scary. How do we convince our package maintainers to do more work?

We recognize that peer review can feel daunting, especially for those from academic backgrounds. Our review process diverges from conventional methods. We see review as a constructive dialogue involving the maintainer team, reviewers, and an editor, all collaborating to enhance the package’s quality and user-friendliness. Occasionally, our editors and reviewers even create issues and pull requests. Our community is readily available to address technical inquiries.

Although participating in a review demands extra time and effort, the benefits are substantial. Improving packaging infrastructure, documentation, and code makes your package more accessible and welcoming to contributors. Over time, this investment will yield significant rewards.

All our reviews are open on GitHub, ensuring transparency. Everyone participating in the peer review process is also expected to adhere to our code of conduct.