Code and Data Review at JCLS

General remarks

In JCLS’s review process, there is a basic and an extended code and data review.

The basic review is mandatory and is part of the normal reviewing process. Basic code quality is a requirement for acceptance. The goal of the basic code review is to make sure that code and data are readily available and can be inspected, beyond what is possible in the text or appendices of a paper.
The extended review is an optional step that authors can request for their article once it has been accepted in principle (e.g. for the annual conference). The goal of the extended code review is to help authors bring the code and data repositories accompanying their paper to a level where their work becomes truly replicable. This means making it as easy as possible for others to repeat the entire data analysis process using the data and code provided by the author(s) of a paper. If the author(s) of a paper under review decide to take advantage of the extended code and data review, a reviewer dedicated specifically to code reviewing will be assigned to the paper and will make proposals for how to revise the code and data repository.

Upon successful completion of the review, the corresponding code and data repository (CDR in short) will receive a certification notice and a badge showing the successful (basic or extended) code and data review.

Basic Code Review Criteria

The following criteria are pertinent for the basic code review. Regardless of the reviewing option they choose, all authors are invited to consider all the criteria when designing their code and data. The basic code review criteria are intended to ensure the findability and long-term availability of the data and code accompanying a paper, ensure basic code quality as well as provide some documentation on the structure and ways to use the data and code.

The URL of the CDR has been correctly included in the paper. (Before acceptance, this repository is anonymized. After acceptance, the regular repository is used.)
There is an archived copy of the CDR, e.g. on Zenodo or Figshare, and its DOI is mentioned in the paper. (This is checked after acceptance.)
The key datasets, scripts and outputs (such as figures and tables) are available in the CDR, or are freely available from places linked in the CDR.
There is a README at the top level of the repository that clearly states the purpose, structure and contents of the repository, including (after acceptance) a reference to the article the repository pertains to. This should be updated after publication with the paper's DOI.
The requirements for the code to run are clearly stated in the README and documented, as far as possible, in a machine-readable manner. For example, a requirements.txt file is used to document the packages that are dependencies of the code.
The repository is usefully structured into folders, in particular separating input data, data or results generated by the code, the code itself, and any visualizations.

Extended Code Review Criteria

The following criteria are pertinent for the extended code review, and are checked on top of the basic code review criteria. Regardless of the reviewing option they choose, all authors are invited to consider all criteria when designing their code and data.

Data availability

All data required for running the scripts is provided in the CDR itself and can be found in dedicated folders, or is freely available from places linked in the CDR. Similarly, all data produced by the scripts is available.
If any additional datasets, tools or other resources are required for running the code in the CDR, their URLs and (if available) DOIs are mentioned in the top-level README. (This is checked after acceptance.)
If data cannot be shared within the CDR itself because the files are too large, solutions like Large File Storage on Github or an external deposit location, like Zenodo or Figshare, should be used. Links to the external files need to be provided in the top-level README.
If data cannot be shared because of copyright issues, two solutions are feasible: Either the materials are transformed to a derived format that is sufficiently information-rich for the analysis to still work (e.g. document-term-matrix for stylometric experiments). Alternatively, the full copyrighted materials are deposited on Zenodo under the “restricted access” modalities.

Documentation

The README explains how to proceed in order to reproduce the key results of the paper and includes, if applicable, a statement of any limitations to the replicability of the code.
Each script in the repository has a short description of its purpose, strategy and requirements. Each function in each script has a short description of its purpose. Use docstrings and comments as appropriate.

Interoperability

The requirements for the code to run are clearly stated in the README and documented, as far as possible, in a machine-readable manner. For example, a requirements.txt file is used to document the packages that are dependencies of the code.
The code is written in such a way as to allow executing it on computers running different operation systems with minimal adjustments. For example, relative paths and ‘os.path.join’ should be used for file paths in Python.
The code is written in such a way as to manage file encoding issues as best as possible. For instance, the file encoding are Unicode / UTF-8 whenever possible and the code makes it explicit that files are to be opened with this encoding.
(optional) The code and data are made available either using a live environment (such as Binder) or by packaging them up entirely (e.g. as a Docker image including all dependencies and requirements).

Replication support

A mapping from the tables and/or figures in the paper to the paths of the corresponding matrices, files and/or figures in the CDR is provided in the README. (This mapping will need to be checked again at the copyediting stage.)
Based solely on the input data provided in the repository (or in other places linked to in the README) and on the code, all results available in the repository can be generated again.
Any settings that users performing a replication of the research will need to adapt in order to run the code (e.g. base paths), or may want to adapt in order to test the influence of certain parameters (e.g. clustering or classification settings) are separated out from the body of the code as variables and clearly marked as such (e.g. by comments).

General code quality

The code is usefully structured into scripts and functions, creating an appropriate level of modularity.
The code is written in such a way as to automatically create all folders and subfolders required to save results to file. No manual creation of folders should be necessary.
The scripts, functions and variables have names that are concise, meaningful and readily understandable. For instance, they avoid excessive use of abbreviations as well as purely functional names (such as ‘dataframe’).
The code follows the relevant coding style conventions, such as PEP8 for Python or tidyverse for R.