ed for which an experiment can be performed and
the results analyzed. Good experimental design utilizes clear, detailed protocols and mechanisms for
observations of phenomena. The analysis of those
observations should be straightforward, with the
desired outcome being evidence for or against the
hypothesis. Science, and AI in particular, has become
very biased toward positive evidence — evidence that
supports our hypotheses. We’ve all been there: when
we find that evidence, we stop looking.
In a bet, the bettors make mutually exclusive
hypotheses. When evidence on one side of the bet is
presented, the adversary naturally continues the
search. Exactly one of the two hypotheses can be
proven true by evidence, and that evidence will disprove the other hypothesis. A final part is played by
an adjudicator, an objective party (or committee)
who the bettors agree will settle any disputes in
resolving the bet. There are no protocols in a bet, but
the detail and specificity for terms of the bet (“
observations of phenomena”) must be high enough to
ensure this mutual exclusivity, and thus one and only
one of the bettors can win the bet. This process of separation of essential elements of a disagreement from
the frequently accompanying “sexy” but tangential
issues is the core contribution of a rigorous scientific
bet. If created properly, this separation should result
in a bet whose outcome can easily and unambiguously be verified by peer-reviewed adjudication.
As compared to traditional hypothesis generation
and experimental design, scientific bets may require
greater commitment and effort. Bettors must call out
what they believe in more detail than in a simple
hypothesis, because they have to distinguish it from
what they don’t believe, what they do not know, and
of what they are uncertain. Normally, scientists are
required to provide this introspection themselves, but
as humans, it is difficult for us to separate the excitement of proving a hypothesis from the rigor of objective analysis. With bets, this introspection is required
for practical interaction with the adversary and adjudicator. The parties must cooperate with each other,
which in some cases may mean working with someone with whom they have an adversarial relationship.
The AI Bookie
There is no widely used public platform for making
scientific bets in AI, so we have decided to create and
operate just such a platform, with the AI Bookie.
Rather than reinvent wheels, we are inspired by and
borrow rules from Long Bets (for example Lowenstein
2016), an “arena for competitive, accountable predic-
tions … [as] a way to foster better long-term think-
ing.” While Long Bets has a focus on long-term bets,
the AI Bookie welcomes bets of any duration on the
future of AI and will exist to foster clearer thinking
about the science around AI in general. The bets will
be documented online (at sciencebets.org) and also
regularly in this column.
We encourage you to find a partner with whom
you have a scientific disagreement and make a bet.
Contact us if you would like help finding an adver-
sary with whom you can collaborate. Once you have
a collaborator and an idea for a bet, we will help you
through the process of crafting a bet to a sufficient
level of precision and rigor.
Over time, the history of well-written bets may act
as a new type of scientific record that has a distinct
role in the scholarly publication process. To this end,
we hope that members of the AI community and of
the scientific community at large will create bets that
have lasting value.
Horvitz, E., and Selman, B. 2012. Interim Report from the
Panel Chairs: AAAI Presidential Panel on Long-Term AI
Futures. In Singularity Hypotheses: A Scientific and Philosophi-
cal Assessment, edited by A. Eden; J. Moor; J. Søraker; and E.
Steinhart, 301–308. Berlin: Springer.
Lipton, Z., and Steinhardt, J. 2018. Troubling Trends in
Machine Learning Scholarship. arXiv preprint. arX-
iv:1807.03341v2 [ stat.ML]. Ithaca, NY: Cornell University
Lowenstein, R. 2016. Why Buffett’s Million-Dollar Bet
Against Hedge Funds Was a Slam Dunk. Fortune Magazine,
Marcus, G.; Rossi, F.; and Veloso, M. 2016. Beyond the Turing Test. AI Magazine, 37( 1): 3–4. doi.org/10.1609/aimag.
Müller, V., and Bostrom, N. 2016. Future Progress in Artificial Intelligence: A Survey of Expert Opinion. In
Fundamental Issues of Artificial Intelligence, edited by V. Müller, 553–
570. Berlin: Springer International Publishing.
Winston, P. 2012. The Next 50 Years: A Personal View.
Biologically Inspired Cognitive Architectures 1(July): 92–99. doi.
Kurt Bollacker is the digital research director at The Long
Praveen Paritosh is a senior research scientist at Google,
Chris Welty is a senior research scientist at Google, Inc.