Table 1. Description of All Variables and Their Factors.
Factor Variable Description
Method Problem Is there an explicit mention of the problem the research seeks to
Objective Is the research objective explicitly mentioned?
Research method Is there an explicit mention of the research method used (empirical,
Research questions Is there an explicit mention of the research question(s) addressed?
Pseudocode Is the AI method described using pseudocode?
Data Training data Is the training set shared?
Validation data Is the validation set shared?
Test data Is the test set shared?
Results Are the relevant intermediate and final results output by the AI
Experiment Hypothesis Is there an explicit mention of the hypotheses being investigated?
Prediction Is there an explicit mention of predictions related to the
Method source code Is the AI system code available open source?
Hardware Is the hardware used for conducting the experiment specified?
Are software dependencies specified?
Experiment setup Are the variable settings shared, such as hyperparameters?
Is the experiment code available open source?
example, Henderson et al. (2017) show that problems
due to nondeterminism in standard benchmark environments and variance intrinsic to AI methods
require proper experimental techniques and reporting procedures. Acknowledging these difficulties,
computational research should be documented properly so that the experiments and results are clearly
The AI research community should strive to facili-
tate reproducible research, following sound scientific
methods and proper documentation in publications.
Concomitant with reproducibility is open science,
which involves sharing data, software, and other sci-
ence resources in public repositories using permissive
licenses. Open science is increasingly associated with
FAIR principles to ensure that science resources have
the necessary metadata to make them findable, acces-
sible, interoperable, and reusable (Wilkinson et al.
2016). Modern digital scholarship promotes proper
credit to scientists who document and share their
research products through citations of datasets, soft-
ware, and innovative contributions to the scientific
The focus in this article is on best practices for documentation and dissemination of AI research to facilitate reproducibility, support open science, and
embrace digital scholarship. We begin with an analysis of recent AI publications that highlights the limitations of their documentation in support of reproducibility.
State of the Art: How AI Research Is
Gundersen and Kjensmo (2018) analyzed how well
empirical AI research is documented to facilitate
reproducibility. Empirical AI research involves evalu-
ating how well computational AI methods solve a
problem. An AI method refers to an abstract method
for solving such problems. Examples include agent
systems that perform collaborative tasks and neural
network architectures trained using backpropagation.