Finding Reviewers#

Sometimes, the most challenging part of our pyOpenSci software peer review process is finding reviewers. This page provides tips and tricks to help you find the right people to review a scientific Python package.

Where to Look for Reviewers?#

As an editor, you can find reviewers through:

When the above doesn’t work and you still need to find a reviewer:

  • Check our reviewer sign-up spreadsheet for names of people who have already volunteered to review for pyOpenSci. The link for this document is pinned to the pyOpenSci Slack #private-editorial-channel.

  • Ping other editors for ideas.

  • Post in the pyOpenSci Slack #software-review channel.

  • Look for users of the package or the data source/upstream service the package connects to (via their opening issues in the repository, starring it, citing it in papers, talking about it on social media).

  • Search for authors/maintainers of related packages on PyPI.

  • Coordinate with our Community Manager to post a call for reviewers on our pyOpenSci social channels.

Important

If you do a reviewer search, please be sure to have new reviewers first sign up using our reviewer signup form.

Criteria for Choosing Reviewers#

Here are the key criteria to consider when selecting a reviewer. You might need to piece this information together by searching PyPI, Conda / Conda-forge and the potential reviewer’s GitHub page and general online presence (personal website, social media profiles).

  • Has not reviewed a package for us within the last 6 months.

  • Some package development/contribution experience.

  • Some domain experience in the field of the package or data source.

  • No conflicts of interest.

Standard reviews should have at least two reviewers; publication fast-track reviews may proceed with one.

  • One reviewer should have expertise in both Python and the scientific domain related to the package submitted.

  • The second reviewer can be more generally focused on the package’s usability, accessibility, and packaging infrastructure. This allows people who are newer to review to get involved while also providing a critical usability check for the package.

A review consisting of a domain expert and a Pythonista is ideal as it provides two distinct perspectives for review. Furthermore, it can often be challenging to find individuals with both the specific domain expertise and packaging expertise.

Timeline for finding reviewers#

If you can, try to find two people to serve as reviewers within two weeks of responding to the issue as the editor. If it takes longer, as often does, make a point to keep the author posted on the issue as you continue your search. You may add language such as:

Hey, @authorGithubHandle I just wanted to drop in to let you know that I’m searching for reviewers for your package. It may take a bit more time.

This type of communication just lets the author know that the process is moving forward. Even if it takes longer to find reviews, authors generally appreciate the communication and understand it’s a volunteer-led process.

Reviewer Diversity Should Be Prioritized#

Try to balance your sense of the potential reviewer’s experience against the complexity of the package.

  • Diversity: Try to find reviewers from different backgrounds and gender identities. For example, if you have two reviewers, only one should be a cis white male.

  • Openness - reviewers should also have demonstrated interest in open source or Python community activities, although blind emailing is fine.

Standard package reviews should have two reviewers, though publication fast-track reviews may proceed with one. Although it is fine for one of them to have less package development experience and more domain knowledge, the review should not be split into two parts. Both reviewers need to review the package comprehensively, from their particular perspectives. If possible, at least one reviewer should have prior reviewing experience, and of course, inviting one new reviewer expands our pool of reviewers.

For standard two-reviewer workflows, at least one reviewer should have some subject matter expertise associated with the package functionality. It is ok and even welcome if one reviewer has more technical knowledge and the other focuses on usability and is less technical. Read through the Guidelines for Reviewers Section to learn more about finding and selecting reviewers.