can be used by competitors to reduce a company’s
This study suggests that industry researchers are
eight times more willing to share data than code.
Why this is the case is not clear. One reason could
be that the data shared is already open data. Investigating this is potential future work, as well as finding
out how to ensure that industry and academic research, accepted at the same conference, will have the
same quality of documentation.
The authors thank Sigbjørn Kjensmo, who collected
most of the data while working on his MS thesis at the
Norwegian University of Science and Technology, and
Rune Havnung Bakken, who read and commented on
a draft of the paper. This work has been carried out at
the Norwegian Open AI Lab at the Norwegian Univer-
sity of Science and Technology, Trondheim, Norway.
4. github.com/kireddo/Standing on the Feet of Giants.
Botvinick, M.; Barrett, D. G.; Battaglia, P.; De Freitas,
N.; Kumaran, D.; Leibo, J. Z.; Lillicrap, T.; Modayil, J.;
Mohamed, S.; Rabinowitz, N. C., Rezende, J., Santoro, A.,
Schaul, T., Summerfield, C., Wayne, G., Weber, T., Wierstra, D.,
Legg, S., and Hassabis, D. 2017. Building Machines That
Learn and Think for Themselves: Commentary on Lake
et al., Behavioral and Brain Sciences, 2017. arXiv.org arXiv:
de Weerdt, M. M.; Gerding, E. H.; Stein, S.; Robu, V.; and
Jennings, N. R. 2013. Intention-Aware Routing to Minimise Delays at Electric Vehicle Charging Stations. In Joint
Proceedings of the Workshop on AI Problems and Approaches
for Intelligent Environments and Workshop on Semantic Cities,
57. New York: Association for Computing Machinery. doi.
Drummond, C. 2009. Replicability Is Not Reproducibility:
Nor Is It Good Science. Paper presented at the Evaluation
Methods for Machine Learning Workshop at the 26th International Conference on Machine Learning (ICML). www.
Goodman, S. N.; Fanelli, D.; and Ioannidis, J. P. A. 2016. What
Does Research Reproducibility Mean? Science Translational Med-
icine 8(341): 341ps12. doi.org/10.1126/scitranslmed.aaf5027.
Gundersen, O. E.; Gil, Y.; and Aha, D. 2018. On Reproduc-
ible AI — Towards Reproducible Research, Open Science,
and Digital Scholarship in AI Publications. AI Magazine
39( 3): 56–68. doi.org/10.1609/aimag.v39i3.2816.
Gundersen, O. E., and Kjensmo, S. 2018. State of the Art:
Reproducibility in Artificial Intelligence. In Proceedings of
the Thirty-Second AAAI Conference on Artificial Intelligence.
Palo Alto, CA: Association for the Advancement of Artifi-
Henderson, P.; Islam, R.; Bachman, P.; Pineau, J.; Precup,
D.; and Meger, D. 2018. Deep Reinforcement Learning That
Matters. arXiv.org arXiv:1709.06560.
Lample, G., Ott, M., Conneau, A.; Denoyer, L.; and Ranzato,
M. 2018. Phrase-Based & Neural Unsupervised Machine
Translation. arXiv.org arXiv:1804.07755.
Mannarswamy, S., and Roy, S. 2018. Evolving AI From
Research to Real Life — Some Challenges and Suggestions.
Paper presented at the Twenty-Seventh International
Joint Conference on Artificial Intelligence (IJCAI), Evolution of the Contours of AI. 5172–79. doi.org/10.24963/
Ott, M.; Auli, M.; Granger, D.; and Ranzato, M. 2018. Analyzing Uncertainty in Neural Machine Translation. Paper
presented at the 35th International Conference on Machine
Learning (ICML) 2018. arXiv.org arXiv:1803.00047.
Peng, R. D. 2011. Reproducible Research in Computational
Science. Science 334(6060): 1226–7.
Salisbury, E.; Kamar, E.; and Morris, M. R. 2018. Evaluating
and Complementing Vision-to-Language Technology for
People Who Are Blind With Conversational Crowdsourcing. In Proceedings of the Twenty-Seventh International
Joint Conference on Artificial Intelligence (IJCAI), 5349–53.
Sculley, D.; Snoek, J.; Wiltschko, A.; and Rahimi, A. 2018.
Winner’s Curse? On Pace, Progress, and Empirical Rigor.
Paper presented at the Sixth International Conference on
Learning Representations (ICLR) 2018 Workshop, Vancouver, BC Canada, April 30–May 3.
Sethi, A.; Sankaran, A.; Panwar, N.; Khare, S.; and Mani, S.
2018. Dlpaper2code: Auto-Generation of Code From Deep
Learning Research Papers. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, 7339–
46. Palo Alto, CA: Association for the Advancement of
Silver, D.; Schrittwieser, J.; Simonyan, K.; Antonoglou, I.;
Huang, A.; Guez, A.; Hubert, T.; Baker, L.; Lai, M.; Bolton,
A., Chen, Y.; Lillicrap, T.; Hui, F.; Sifre, L.; van den Driessche,
G.; Graepel, T.; and Hassabis, D. 2017. Mastering the Game
of Go Without Human Knowledge. Nature 550(7676): 354–9.
Stodden, V. C. 2011. Trust Your Science? Open Your Data
and Code. Amstat News 2011(July): 21–2. web.stanford.
Odd Erik Gundersen (PhD, Norwegian University of Science and Technology) is the Chief AI Officer at the renewable energy company TrønderEnergi AS and an Adjunct
Associate Professor at the Department of Computer Science
at the Norwegian University of Science and Technology.
Gundersen has applied AI in the industry, mostly for startups,
since 2006. Currently, he is investigating how AI can be applied in the renewable energy sector and for driver training,
and how AI can be made reproducible.