Health), and industry (such as IBM and Microsoft)
are examples. Only eight of the papers in the collaboration group are from collaborations between
private research and government institutions. In
this study, I present the results from collaboration
studies and industry studies as well, despite small
Table 2 presents the mean values for the eight variables comprising the factor Method for each group
of papers. Industry scores highest on the variables
Problem description, Goal, and Experiment setup,
while the combination (C + I) of collaborations and
industry have the same score as academic for Problem
description. Academic research scores higher than
industry, collaboration, and the combination for
Research method, Research question, Pseudo code,
and Prediction. None of these results are statistically
significant. Academic research also scores highest on
Hypothesis, and this is statistically significant.
Table 3 presents the mean values for the four var-
iables comprising the factor Data for each of the
groups of papers. Academic research has the highest
score for Training data. The result for this variable
is statistically significant when compared with in-
dustry and the combination. Academia also has the
highest score for Validation data and Test data as
well, but these results are not statistically significant.
Industry has the highest score for Results, and C + I
has a lower score than academia. None of these find-
ings are statistically significant.
Table 4 presents the mean values for the four variables comprising the factor Experiment for each
of the groups of papers. Academic research scores
highest on Hardware specification, and this result is
statistically significant when compared with C + I.
Industry has the best score on Method code, Experiment code, and Software dependencies. However,
the confidence is low as the error is very high. The
scores for C + I are lower for all these variables when
comparing to academic research.
Figure 4 shows three spider plots of the mean for
the variables of each of the three factors for all the
Factor Variable Description
Problem Is there an explicit mention of the problem the research seeks to solve?
Objective Is the research objective explicitly mentioned?
Research method Is there an explicit mention of the research method used (empirical, theoretical)?
Research questions Is there an explicit mention of the research question(s) addressed?
Pseudocode Is the AI method described using pseudocode?
Hypothesis Is there an explicit mention of the hypotheses being investigated?
Prediction Is there an explicit mention of predictions related to the hypotheses?
Experiment setup Are the variable settings shared, such as hyperparameters?
Training data Is the training set shared?
Validation data Is the validation set shared?
Test data Is the test set shared?
Results Are the relevant intermediate and final results output by the AI program shared?
Method source code Is the AI system code available open source?
Experiment source code Is the experiment code available open source?
Software dependencies Are software dependencies specified?
Hardware Is the hardware used for conducting the experiment specified?
Figure 2. Method, Data, and Experiment, and the Variables That Specify Them.