An Overview Of the Peer Review Process#

pyOpenSci open peer review steps#

There are several components to the pyOpenSci peer review process. Below, we overview the entire process from start to finish.

Step 1 optional: Author submits pre-submission inquiry:*#

A presubmission inquiry is wise to do if you are unsure of whether your package is in scope. To submit a pre-submission inquiry, open up an issue using the presubmission template in our pyopensci/software-review repository. During this time an Editor in Chief will review for scope and performs basic checks for package infrastructure

  • Estimated time: ~1-2 weeks

These are the basic checks that your package should have prior to being submitted for peer review

  • [ ] Fit: The package meets criteria for fit and overlap.

  • [ ] Documentation The package has sufficient documentation available online (README, sphinx docs) to allow us to evaluate package function and scope without installing the package. This includes:

    • [ ] Get started vignettes or tutorials that help a user understand how to use the package and what it can do for them

    • [ ] API documentation - this includes clearly written doc strings with variables defined using a standard docstring format

    • [ ] README that clearly articulates the function of the package

    • [ ] Contributing documentation that details how to install a development environment and how to contribute to the package

  • [ ] Issue Header Information: All of the required issue header information is contained within the top of the issue.

  • [ ] Automated tests: Package has a testing suite and is tested via GitHub actions or another Continuous Integration service.

  • [ ] License: The package has an OSI accepted license

  • [ ] Repository: The repository link resolves correctly

  • [ ] README: The package has a README with clear explanation of what the packages does and instructions on how to install it along with development instructions.

  • [ ] Contributing statement: The package has a contributing.md file that details how to contribute to the package.

  • [ ] Package overlap: That package doesn’t fully overlap with functionality of other packages that have already been submitted to pyOpenSci

  • [ ] Archive (JOSS only, may be post-review): The repository DOI resolves correctly

  • [ ] Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

2. Author submits their package for review#

To do this, you open an issue using the software submission template in our pyopensci/software-review repository.

3. Editor in Chief reviews package submission#

The Editor in Chief will review your submission at this point for both package scope and minimal infrastructure criteria (listed above)

  • TIME ~2 weeks (or longer if editor requests changes that take the author longer to implement)

3. Editor finds reviewers for package#

At this point if your package has the minimal infrastructure requirements and is in scope the Editor in Chief will assign an editor to review your package. That editor will then look for reviewers.

Time: ~ 2-3 weeks

4. Once we had an editor and 2 reviewers on board, the review begins#

Reviewers have 3 weeks to return a review. To do this they will use our reviewer template in the reviewer guide and past that, filled out, into the issue.

TIME: ~3 weeks

5. Author response to reviews#

At this point the authors should respond to the review. We prefer that authors respond within 2 weeks of the submitted review. We also understand that it may take longer to actually implement the changes requested in the review. But we request that authors respond to reviews to acknowledge that they have seen them.

The reviewers are encouraged to open pull requests and issues in to help the maintainers update their package.

6. Often there is some back and forth between reviewers and maintainers at this step#

There is no specified duration for this period. Rather as long as all parties are responsive within 2 weeks the review shall continue until the author has completed work addressing the reviews.

7. Package acceptance#

Once the maintainers have completed updating the package, the assigned editor will ask the reviewers if they are happy with changes made. At this point the editor performs one last check on the package. And accepts it if that is appropriate.

Now, there are a few final cleanup activities including adding a pyOpenSci badge to the package README file, and creating a new release on GitHub from the reviewed version. The package is now accepted into the pyOpenSci ecosystem!

JOSS submission#

JOSS refers to the Journal of Open Source Software. If the maintainer wishes, and their package is within JOSS’ scope, they can now be fast tracked through the JOSS review process (another review is not required for this step).

If you want to learn more about each step listed above, we suggest that you read through the guides below:

  • Authors Guide: If you are a package maintainer who is interested in submitting a package to pyOpenSci, check out the authors guide.

  • Reviewers guide: If you have volunteered to be a reviewer for a pyOpenSci package, you will want to carefully read through the reviewer guide for guidance on how we run our reviews. Thank you in advance for volunteering your time to support open science and open source software!

  • Editor guide: Editors for pyOpenSci have previous experience reviewing packages for pyOpenSci. The editor guide will walk you through the best practices for the editor role.

  • Editor in Chief guide: The Editor in Chief is a rotating position within pyOpenSci held by members of the pyOpenSci editorial board.