We Can Reward Reproducibility
As reproducibility of research is a cornerstone of science, reproducibility should be rewarded in the review
process and when assessing for scientific positions.
When it comes to reproducibility, academia could
actually learn from industry — not necessarily from
industry research practices, but from the software
engineering practices that the industry follows. Software engineers focus on building quality software
and continuously evaluating its performance. Software development methodologies including Agile
(such as Scrum and Kanban), test-driven development, and code reviews have been developed to help
increase the quality of the software. The reason is
that the performance of the software is directly related to how well the companies themselves perform
(and return financial investment), so reproducibility
is a key concern together with proper performance
evaluation. For companies that develop AI and machine learning software, this diligence in evaluating
software extends to the AI and machine learning
Microsoft IBM Baidu Didi Facebook Others
Figure 9. The Tech Giants Microsoft, IBM, Baidu, Didi, and Facebook Published 32 of 57 Papers in the Group C + I.
The total number of companies does not add to 57, as some papers have authors from more than one company.
software. Versioning of code and data are required
to ensure the capability of monitoring performance
In science, reproducibility is key for ensuring that
our beliefs regarding a concept, such as an AI program,
are correct. It is through building and organizing the
set of these beliefs that we expand our knowledge. As
scientists, we should optimize for advancing knowledge. Therefore, we should ensure that our results
are correct, which means that we must be able to reproduce our own results while enabling independent
researchers to do the same. As discussed above, the
incentives for individual scientists are not necessarily aligned for this right now, and we need an open
discussion on what can be changed to get there.
For companies, maintaining a competitive advantage is important and sharing could enable competitors to close the gap. Hence, all openness can be
considered a net win for the AI research community.
The fact that companies share methods, code, and
data should be applauded. However, given that there