Review Process

This page walks through the process of leading a package review for pyOpenSci.

Editor checklist: Get Started With Leading a Package Review

Follow the checklist below when serving as an editor for a package submitted to pyOpenSci for review.

All reviews happen in GitHub issues. The template for the yaml header of a review submission below will be referenced multiple times in the steps below:

---
name: Submit Software for Review
about: Use to submit your Python package for pyOpenSci peer review
title: ''
labels: 1/editor-checks, New Submission!
assignees: ''
---
Submitting Author: Name (@github_handle)
All current maintainers: (@github_handle1, @github_handle2)
Package Name: Package name here
One-Line Description of Package: Description here
Repository Link:
Version submitted:
Editor: TBD
Reviewer 1: TBD
Reviewer 2: TBD
Archive: TBD
Version accepted: TBD
Date accepted (month/day/year): TBD
---

✔️ 1. First, tag the submission issue on GitHub & assign yourself to the issue

Once you begin the review process as an editor:

• Tag the submitted GitHub issue with the 1/editor-checks tag if it hasn’t already been tagged by the editor-in-chief.

• Make sure that you are assigned to the issue on GitHub (ie, your name is on the right-hand side of the issue as the person running it).

Make sure that your name is both listed in the YAML at the top of the issue and also that you are assigned to the issue on GitHub (on the right-hand side of the issue).

• Check the YAML template at top of the submitted GitHub issue, make sure that mandatory parts of the template are filled out.

• If elements are incomplete, direct the authors toward filling in any missing pieces.

Submitting Author: SUBMITTING AUTHOR NAME HERE (Name @github_handle)
All current maintainers: ALL CURRENT MAINTAINERS LISTED HERE (Name @github_handle1, Name @github_handle2)
Package Name: PACKAGE-NAME-HERE
One-Line Description of Package: DESCRIPTION OF THE PACKAGE HERE
Repository Link: REPO-LINK
Version submitted: VERSION-SUBMITTED

Editor in Chief checks for structure & scope should be completed first

The editor in chief who initially engaged with this review should have already evaluated the package-level Editor Checks section for Fit, Automated Tests, Documentation, License, and Repository.

They also should have checked whether the package is in scope for pyOpenSci. And whether there is functionality overlap with functionality of any other existing Python packages.

However, in some instances, the editor-in-chief may request that an editor perform these tasks. Be sure to check the issue to ensure the above checks have been implemented before initiating the review.

If the package does not fit the pyOpenSci scope and policies and needs to be rejected, see this section in the editor-in-chief guide about how to respond.

✔️ 2. Respond to the submitter in the GitHub issue

Once the above is complete, you are ready to add an editor response to the issue. This step ensures that the package is ready to be reviewed. It also ensures that we are using our volunteer reviewer time effectively.

• Add a comment to the issue that contains an exact copy of the Editor Response template (see below) filled out with your response to the checks that begin the review.

• Change the label of the issue to 2/seeking-reviewer(s)

• Note that, at this stage, you likely do not have reviewers yet!

## Editor response to review:

---

## Editor comments
<Add additional banter here that you wish..>
<In step 3 you will add a welcome message to the reviewers>


### Please fill out our pre-review survey
Before beginning your review, please fill out our [pre-review survey](https://forms.gle/F9mou7S3jhe8DMJ16). This helps us improve all aspects of our review and better understand our community. No personal data will be shared from this survey - it will only be used in an aggregated format by our Executive Director to improve our processes and programs.
- [ ] reviewer 1 survey completed.
- [ ] reviewer 2 survey completed.
- [ ] reviewer 3 (if applicable)


The following resources will help you complete your review:

1. Here is the **[reviewers guide](https://www.pyopensci.org/software-peer-review/how-to/reviewer-guide.html)**. This guide contains all of the steps and information needed to complete your review.
2. Here is the **[review template](https://www.pyopensci.org/software-peer-review/how-to/reviewer-guide.html#peer-review-template)** that you will need to fill out and submit
here as a comment, once your review is complete.

Please get in touch with any questions or concerns! Your review is due: <Insert deadline DATE HERE>
---

Reviewers:
Due date:

Important

Important: If in the initial checks you find any major gaps in the package’s structure, request changes before assigning reviewers.

Note

Authors may opt to submit to JOSS after a successful PyOpenSci submission. If this is the case, please note that they are not required to have the paper.md file before the JOSS submission.

✔️ 3. Identify reviewers for the Python package

Each review needs reviewers assigned before it can proceed.

• Standard reviews should have at least two reviewers, one with Python and scientific domain expertise, and one with the knowledge of package usability, accessibility, and packaging infrastructure.

• Packages eligible for the publication fast-track pathway may proceed with one reviewer.

• Find reviewers within two weeks of responding to the issue as the editor.

• See Finding Reviewers for tips on where to look and template language for keeping the author up-to-date.

Diversity in the editorial & reviewer team is important

Diversity is core to the pyOpenSci mission. As such, it’s important to have an editorial team with diverse backgrounds. Standard reviews usually involve an editor plus two reviewers, while publication fast-track reviews may use one reviewer.

In your search for reviewers, please ensure that there is diversity in the team supporting package review. Reviewers should have different backgrounds and different gender identities whenever possible. pyOpenSci supports mentoring new reviewers if needed!

• Once you have assigned reviewers to the review, you will update the editor response above with:

1. Reviewer GitHub handles and

2. The review deadline date.

Tip

At this point, you can add the following to the Editor Response to Review under Editor Comments: :wave: Hi @reviewer-one and @reviewer-two! Thank you for volunteering to review for pyOpenSci!

• Change the label on the issue to 3/reviewer(s)-assigned

Warning

Make sure to ask the reviewers for their preferred means of contact or a reliable way to get in touch with them.

✔️ 4. Onboard reviewers

See onboarding reviewers to learn more about inviting reviewers and providing support.

Once reviewers have been identified:

• Tag issue with 3/reviewer(s)-assigned tag.

• Add reviewer names and review due date to the Editor Comment that you left above in step 2.

• Also add the reviewer names to the YAML template at the very top of the issue.

Editor: Full Name (@github_username)
Reviewer 1: Full Name (@github_username)
Reviewer 2: Full Name (@github_username)
Archive: Filled out when the review is complete.
Version accepted: Filled out when the review is complete.

Editor responsibilities during the review

During the review process, it is important to check in with the reviewers to ensure that things are moving smoothly:

• Check in with reviewers and authors occasionally. Offer clarification and help as needed.

• Aim for ~3 weeks for review, 2 weeks for subsequent changes, and 1 week for reviewer approval of changes.

• If a review has not been submitted after 2 weeks, ping the reviewer(s) within the review issue to ensure they are aware of the 3-week deadline.

• If you are waiting for a maintainer to respond to you, be sure to add the label pending-maintainer-response to the issue.

Managing labels throughout the review process

As the review progresses through different stages, you should update the labels to reflect the current state of the review. Remove older labels and keep only the label(s) that represent the most recent step in the review process.

For example:

• When you complete the editor checks and begin seeking reviewers, remove 1/editor-checks and add 2/seeking-reviewer(s).

• When reviewers are assigned, remove 2/seeking-reviewer(s) and add 3/reviewer(s)-assigned.

• When reviews are submitted, remove 3/reviewer(s)-assigned and add 4/review-in-awaiting-changes.

• If the review is put on hold, add the holding label. You can keep other status labels to indicate where in the process the review was paused.

This approach keeps the issue labels clean and makes it easy to see at a glance what stage each review is currently in.

✔️ 5. What to do when reviews are in

• Once all reviews are submitted, change the review status tag to 4/review-in-awaiting-changes.

If the author stops responding, refer to the policies and/or ping the other editors in the Slack channel <Not available publicly yet> for discussion.

Once the author has responded to the reviews and made appropriate changes:

• Briefly check to ensure that the changes were indeed made.

• Change the issue status tag to 5/awaiting-reviewer-response.

Important

If you need to stop the review and close the issue for some reason during the process, please be sure to let all parties know (the maintainer and reviewers) before closing the issue.

Be sure to:

• explain why the decision was made

• thank them for their work

• make a note to assign the reviewer to another submission with a high chance of smooth review next time (for example, a package author who has already submitted packages to us).

What to do if reviewers become quiet

In some cases, the reviewers get busy and stop responding to the issue. If a reviewer hasn’t responded to a direct ping @reviewer-user-name on the issue, try to email them (if an email is available). If email doesn’t work and they haven’t responded in over a month, do the following:

1. If they have already submitted a review, you can move the review forward by making sure the review comments were addressed by the author yourself.

2. If the reviewer has not yet submitted a review on the issue, leave a note on the review that you will look for a new reviewer. When you find a new reviewer, replace the old reviewer’s name with the new reviewer’s name in the YAML at the top of the issue.

In some cases, you may be able to find an editor or reviewers in the pyOpenSci Slack community that can help move the review forward.

Putting a review on hold & handling non-responsive authors

In some cases, an author may need more time to respond to review comments. In this case, the author can choose to have their submission put on hold. As an editor, you should apply the holding label to the GitHub issue.

The holding status will be revisited every 3 months.

After one year, the issue will be closed if there is no movement towards responding to reviews by the author.

If the author is not responding, the editor should:

• Tag the author (@author-github-handle) in an issue comment notifying them that we will close the issue in one month if there is no response.

• Close the issue if two months have passed without a reply.

Before closing the issue, email the author using the email provided in the package metadata file of the package. The email could be in any of the three files in the package: setup.py, pyproject.toml (preferred), or setup.cfg.

Important

If a submission is closed and the author wishes to resubmit, they will have to start a new submission.

If the package is still in scope, the author will have to respond to the first round of reviews first. After that is complete, you can begin looking for new reviewers to evaluate the package. This ensures that none of the review energy spent in the first review goes to waste.

✔️ 6. How to accept a package into the pyOpenSci ecosystem

Once the package has been accepted through the review process:

• Use the approval template below in the issue. This template asks the authors and reviewers to add themselves and the package that was approved to the pyOpenSci website. They can perform this step regardless of the JOSS submission process.

---

🎉 <package-name-here> has been approved by pyOpenSci! Thank you <maintainer-name-here> for submitting <package-name> and many thanks to <reviewer-names-here> for reviewing this package! 😸

## Author Wrap Up Tasks

There are a few things left to do to wrap up this submission:

- [ ] Activate [Zenodo](https://zenodo.org/) watching the repo if you haven't already done so.
- [ ] Tag and create a release to create a Zenodo version and DOI.
- [ ] Add the badge for pyOpenSci peer-review to the README.md of <package-name-here>. The badge should be `[![pyOpenSci Peer-Reviewed](https://pyopensci.org/badges/peer-reviewed.svg)](https://github.com/pyOpenSci/software-review/issues/issue-number)`.
- [ ] Please fill out the [post-review survey](https://forms.gle/BLGVCfUdiJHS5YJY7). All maintainers and reviewers should fill this out.

<IF JOSS SUBMISSION>
It looks like you would like to submit this package to JOSS. Here are the next steps:

- [ ] <Editor Task> Once the JOSS issue is opened for the package, we strongly suggest that you subscribe to issue updates. This will allow you to continue to update the issue labels on this review as it goes through the JOSS process.
- [ ] Login to the JOSS website and fill out the JOSS submission form using your Zenodo DOI. **When you fill out the form, be sure to mention and link to the approved pyOpenSci review.** JOSS will tag your package for expedited review if it is already pyOpenSci approved.
- [ ] Wait for a JOSS editor to approve the presubmission (which includes a scope check).
- [ ] Once the package is approved by JOSS, you will be given instructions by JOSS about updating the citation information in your README file.
- [ ] When the JOSS review is complete, add a comment to your review in the pyOpenSci software-review repo here that it has been approved by JOSS. An editor will then add the JOSS-approved label to this issue.

🎉 Congratulations! You are now published with both JOSS and pyOpenSci! 🎉
<IF JOSS SUBMISSION/>

## Editor Final Checks

Please complete the final steps to wrap up this review. Editor, please do the following:

- [ ] Make sure that the maintainers filled out the [post-review survey](https://forms.gle/BLGVCfUdiJHS5YJY7)
- [ ] Invite the maintainers to submit a blog post highlighting their package. Feel free to use / adapt [language found in this comment](https://github.com/pyOpenSci/software-submission/issues/93#issuecomment-1591687581) to help guide the author.
- [ ] Change the status tag of the issue to `6/pyOS-approved6 🚀🚀🚀`.  (You can remove all other labels; If the the package is moving on to JOSS then add a JOSS label)
- [ ] Update the header of this issue with the date accepted, the version accepted, and the accepted version DOI (from Zenodo) to the header of this issue.
- [ ] Invite the package maintainer(s) and both reviewers to the pyOpenSci Slack.
- [ ]  If the review is complete (not submitting to JOSS), close the issue when all items on this checklist are complete. If it's moving to JOSS, please close the issue once the JOSS review is complete and you've performed the task below.
- [ ] If the author submits to JOSS, please continue to update the labels for JOSS on this issue until the author is accepted (do not remove the `6/pyOS-approved` label). Once accepted add the label `9/joss-approved` to the issue. Skip this check if the package is not submitted to JOSS.
- [ ] If the package is JOSS-accepted please add the JOSS DOI to the YAML at the top of the issue.

---

If you have any feedback for us about the review process please feel free to share it here. We are always looking to improve our process and documentation in the [peer-review-guide](https://www.pyopensci.org/software-peer-review).

• If the package moves on to JOSS - be sure to continue to update the labels to track the JOSS review process (but do NOT remove the 6/pyOS-approved label).

• Update the YAML at the top of the issue with the:

  • version of the package that was approved,

  • Zenodo DOI for the approved version and

  • the date approved.

Archive: UPDATE-THIS-WITH-A-ZENODO-ARCHIVE-BADGE-TBD
Version accepted: UPDATE-THIS-TBD
Date accepted (month/day/year): UPDATE-THIS-TBD

• Once the YAML is updated, change the status tag of the issue to 6/pyOS-approved 🚀🚀🚀`.

  • The listing on the PyOpenSci website is updated by a cron job that parses the YAML in all issues with the tag 6/pyOS-approved 🚀🚀🚀`, so a new package might take a few days to appear; in particular, if there are formatting errors in the YAML that need to be corrected.

  • Editors can manually trigger the cron job and/or check the output of the last run to see if there are any YAML formatting errors.

  • Other “after review” tasks, like the post-review survey, can be done after the package is accepted; the cron job does not depend on them.

Note

• Archive refers to an archive created through a release. You can use Zenodo to create this archive and provide the package with a citable DOI. If Zenodo is used, please add the Zenodo link and/or badge link here.

• Version refers to the final package version that was accepted by pyOpenSci. This is the final version as presented after all feedback from the reviews has been considered and implemented.

✔️ OPTIONAL: Instructions for Submitting to JOSS

If the package fits within the JOSS Scope, once the package has been approved by pyOpenSci:

• Tag the issue 7/under-joss-review.

• Instruct the author to read the paper.md file requirements for JOSS and ensure that paper is added to the repo.

• Direct the package author to follow the instructions to submit the package to JOSS.

These instructions loosely include:

1. Login to the JOSS website and fill out the JOSS submission form. When you fill out the form, be sure to mention and link to the approved pyOpenSci review.

2. Wait for a JOSS editor to approve the presubmission (which includes a scope check).

Important

The scope of packages accepted by pyOpenSci is sometimes different from those accepted by JOSS. Not all pyOpenSci accepted packages will be accepted by JOSS. Further, packages that have been previously published elsewhere may not be eligible to be published with JOSS unless significant changes and improvements to package functionality have been made.

pyOpenSci reviewers are not responsible for reviewing the JOSS paper. Therefore, they are not required to read/review the paper.md file. The JOSS paper review will occur after you submit your package for fast-tracking through JOSS’s review process.

JOSS will accept the pyOpenSci review and direct the author to check their paper.md file. Once JOSS accepts the package, the author will be instructed to add the JOSS DOI badge to their package README file.

Once the package is accepted by JOSS and the DOI badge resolves properly:

• Tag the issue with 9/joss-approved. (Please do NOT remove the pyos-approved label, instead add the JOSS-approved label)

Last Steps Before Closing the Review Issue

Once the review is complete, you can close the issue. Before doing that:

• Be sure that the issue is correctly tagged with 6/pyOS-approved (and 9/joss-approved if authors decided to submit to JOSS and were accepted).

• Check the pyOpenSci website to ensure:

  • The package was added to the pyOpenSci website.

  • Reviewers and maintainers are listed on the contributors page.

  • Make sure the YAML at the top of the issue is filled out and up to date.

• If JOSS approves the package, be sure that the issue is tagged with 7/JOSS-approved and that the archive / DOI information at the top is updated with the JOSS archive before closing the issue.

Congratulations, you have completed a review for pyOpenSci!