Guide for Reviewers#

Thank you for taking the time to review a package for pyOpenSci and for contributing to making open source scientific software easier to use by the community! If you have any questions about the review process, feel free to reach out to one of our editors or to post a question on our discourse forum.

Please be respectful and kind to the authors in your reviews. Our code of conduct is mandatory for everyone involved in our review process.

Not yet a pyOpenSci reviewer? Join us!

We could use your help! Fill out our reviewer form. We will contact you if we have a package that we need reviewers for. It’s OK if you’ve never reviewed a package before! We’ll walk you through it.

Sign-up Here (Google Form)

A guide for new reviewers#

Thank you for considering reviewing for pyOpenSci. We welcome reviewers from a diversity of background and with varying levels of Python technical expertise.

Some of the basic things that we look for in a review include:

  • Familiarity with using the Python programming language.

  • Ability to evaluate a Python package for usability and documentation quality.

  • Ability to provide a technical review of Python package structure and code quality / approach to solving the problems that the package seeks to address.

We like to have a mix of technical and usability focus in our reviews so it’s ok if you don’t have all of the above skills!

New to package review? We offer mentorship!

If you are interested in peer review but have never reviewed before, we offer a mentorship program where we will pair you up with someone who has more experience reviewing code. From this experience you can learn more and empower yourself with code review skills. Software review skills are generally useful in data science, so they are skills worth investing in!

Get Started With Your Review#

As a new reviewer you should start by installing the package that you are reviewing locally to test it out.

To install the package, you can either:

  • fork the package repository and install it in development/ editable mode pip install --editable .

  • or you can install it directly from pip or conda-forge.

Important

Be sure that the version that you are reviewing aligns with the version submitted by the author. The package version can be found at the top of the review issue. In the example below you can see that pyrolite version 0.2.5 was submitted for review.

That is the version that you should install!

Example header of a peer review submission issue on GitHub:

Submitting Author: Name (@morganjwilliams)
Package Name: pyrolite
One-Line Description of Package: A set of tools for getting the most from your geochemical data.
Repository Link: https://github.com/morganjwilliams/pyrolite
Version submitted: 0.2.5
Editor: @lwasser

General Guidelines For Reviewing A Python Package for PyOpenSci#

Note

If you are submitting a review, we appreciate submission within 3 weeks of accepting a review. Please contact the editor directly or in the submission thread to inform them about possible delays.

To review a package, please begin by copying our review template (see below) and using it as a high-level checklist.

Peer Review Template#

## Package Review

*Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide*

- [ ] As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor _before_ starting your review).

#### Documentation

The package includes all the following forms of documentation:

- [ ] **A statement of need** clearly stating problems the software is designed to solve and its target audience in README.
- [ ] **Installation instructions:** for the development version of the package and any non-standard dependencies in README.
- [ ] **Vignette(s)** demonstrating major functionality that runs successfully locally.
- [ ] **Function Documentation:** for all user-facing functions.
- [ ] **Examples** for all user-facing functions.
- [ ] **Community guidelines** including contribution guidelines in the README or CONTRIBUTING.
- [ ] **Metadata** including author(s), author e-mail(s), a url, and any other relevant metadata e.g., in a `pyproject.toml` file or elsewhere.

Readme file  requirements
The package meets the readme requirements below:

- [ ] Package has a README.md file in the root directory.

The README should include, from top to bottom:

- [ ] The package name
- [ ] Badges for:
    - [ ] Continuous integration and test coverage,
    - [ ] Docs building (if you have a documentation website),
    - [ ] A [repostatus.org](https://www.repostatus.org/) badge,
    - [ ] Python versions supported,
    - [ ] Current package version (on PyPI / Conda).

*NOTE: If the README has many more badges, you might want to consider using a table for badges: [see this example](https://github.com/ropensci/drake). Such a table should be more wide than high. (Note that the a badge for pyOpenSci peer-review will be provided upon acceptance.)*

- [ ] Short description of package goals.
- [ ] Package installation instructions
- [ ] Any additional setup required to use the package (authentication tokens, etc.)
- [ ] Descriptive links to all vignettes. If the package is small, there may only be a need for one vignette which could be placed in the README.md file.
    - [ ] Brief demonstration of package usage (as it makes sense - links to vignettes could also suffice here if package description is clear)
- [ ] Link to your documentation website.
- [ ] If applicable, how the package compares to other similar packages and/or how it relates to other packages in the scientific ecosystem.
- [ ] Citation information

#### Usability
Reviewers are encouraged to submit suggestions (or pull requests) that will improve the usability of the package as a whole.
Package structure should follow general community best-practices. In general please consider whether:

- [ ] Package documentation is clear and easy to find and use.
- [ ] The need for the package is clear
- [ ] All functions have documentation and associated examples for use
- [ ] The package is easy to install


#### Functionality

- [ ] **Installation:** Installation succeeds as documented.
- [ ] **Functionality:** Any functional claims of the software been confirmed.
- [ ] **Performance:** Any performance claims of the software been confirmed.
- [ ] **Automated tests:**
  - [ ] All tests pass on the reviewer's local machine for the package version submitted by the author. Ideally this should be a tagged version making it easy for reviewers to install.
  - [ ] Tests cover essential functions of the package and a reasonable range of inputs and conditions.
- [ ] **Continuous Integration:** Has continuous integration setup (We suggest using Github actions but any CI platform is acceptable for review)
- [ ] **Packaging guidelines**: The package conforms to the pyOpenSci [packaging guidelines](https://www.pyopensci.org/python-package-guide).
    A few notable highlights to look at:
    - [ ] Package supports modern versions of Python and not [End of life versions](https://endoflife.date/python).
    - [ ] Code format is standard throughout package and follows PEP 8 guidelines (CI tests for linting pass)

#### For packages also submitting to JOSS

- [ ] The package has an **obvious research application** according to JOSS's definition in their [submission requirements](http://joss.theoj.org/about#submission_requirements).

*Note:* Be sure to check this carefully, as JOSS's submission requirements and scope differ from pyOpenSci's in terms of what types of packages are accepted.

The package contains a `paper.md` matching [JOSS's requirements](http://joss.theoj.org/about#paper_structure) with:

- [ ] **A short summary** describing the high-level functionality of the software
- [ ] **Authors:** A list of authors with their affiliations
- [ ] **A statement of need** clearly stating problems the software is designed to solve and its target audience.
- [ ] **References:** With DOIs for all those that have one (e.g. papers, datasets, software).

#### Final approval (post-review)

- [ ] **The author has responded to my review and made changes to my satisfaction. I recommend approving this package.**

Estimated hours spent reviewing:

---

#### Review Comments

Other review guidelines to consider#

In addition to checking off the minimum criteria specified in the review template, please also provide general comments addressing the following:

  • Is the code well written and efficient?

    • Are there improvements that could be made to the code style?

    • Is there code duplication in the package that should be reduced?

    • Are there performance improvements that could be made?

    • Are functions and arguments named to work together to form a common, logical programming API that is easy to read, and autocomplete?

  • Does the package comply with the pyOpenSci packaging guide](https://www.pyopensci.org/python-package-guide)?

  • Are there user interface improvements that could be made?

  • Is the documentation (installation instructions/vignettes/examples/demos) clear and sufficient?

    • Does the documentation use the principle of multiple points of entry i.e. takes into account the fact that any piece of documentation may be the first encounter the user has with the package and/or the tool/data it wraps?

If you have your own relevant data/problem that you could use the package to solve, work through it with the package. You may find rough edges and use-cases the author didn’t think about.

Important

Some maintainers do not ship the tests with their package. This means that the dependencies for testing may be missing. Check to have at least pytest installed. Also, when cloning the repository to run the tests, please check the cloned version aligns with the submitted one.

Submit your review#

  • When your review is complete, paste it as a comment into the package software-review issue.

  • Additional comments are welcome in the same issue. We hope that package reviews will work as an ongoing conversation with the authors as opposed to a single round of reviews typical of academic manuscripts.

  • You may also submit issues or pull requests directly to the package repo if you choose, but if you do, please comment about them and link to them in the software-review repo comment thread so we have a centralized record and text of your review.

  • Please include an estimate of how many hours you spent on your review afterwards.

Review follow-Up#

Authors should respond within 2 weeks with their changes to the package in response to your review. At this stage, we ask that you respond as to whether the changes sufficiently address any issues raised in your review. We encourage ongoing discussion between package authors and reviewers, and you may ask editors to clarify issues in the review thread as well.

Examples of Past pyOpenSci Reviews#

It might be helpful to consult some previous reviews and read about the experiences of other reviewers. In general you can find submission threads of onboarded packages here. Here are a few chosen examples of reviews (note that your reviews do not need to be as long as these examples):

Lessons Learned from rOpenSci#

You can read blog posts written by reviewers about their experiences via this link. In particular, in this blog post by Mara Averick read about the “naive user” role a reviewer can take to provide useful feedback even without being experts of the package’s topic or implementation, by asking themselves “What did I think this thing would do? Does it do it? What are things that scare me off?”. In another blog post Verena Haunschmid explains how she alternated between using the package and checking its code.

As both a former reviewer and package author Adam Sparks wrote this “[write] a good critique of the package structure and best coding practices. If you know how to do something better, tell me. It’s easy to miss documentation opportunities as a developer, as a reviewer, you have a different view. You’re a user that can give feedback. What’s not clear in the package? How can it be made more clear? If you’re using it for the first time, is it easy? Do you know another R package that maybe I should be using? Or is there one I’m using that perhaps I shouldn’t be? If you can contribute to the package, offer.”

Feedback on the pyOpenSci open peer review process#

We welcome any feedback and/or questions about the review process! We encourage you to post any issues, feedback or questions in our Discourse forum.