Editorial Team: How does one evaluate AI research?
Gil: For me, the evaluation of research in AI depends on the disciplinary angle that is taken. There
are significant differences in evaluation and in methodology for AI research that focus on human computer interaction, or cognitive science, or physics,
or social networking, or philosophy. For example, a
cognitive psychology methodology may place a lot
of emphasis on theoretical models of the mind and
implementing AI systems that approximate them
well based on particular measurements from user
studies. In contrast, a more mathematical methodology may evaluate accuracy based on some statistical metric and may be more focused on algorithmic improvement. So different evaluation methods
are appropriate for different kinds of AI research.
Editorial Team: Is it difficult to learn how to understand and appreciate different disciplinary angles?
Gil: You have to learn enough about the other discipline, which can be a challenge. I’ll let you in on a
little secret: Long plane rides are a great opportunity
to learn something new and different. Whenever I
am on one, I really cherish the opportunity to read
without interruption. Before the trip, I will prepare
a big folder of materials and books to teach myself
about a topic that I am less familiar with. Reading
them makes the trip feel very short and I can learn
something about a new discipline or topic that I have
been curious about!
Editorial Team: What factors do you see inhibiting
the rate of success in AI research?
Gil: We need to emphasize more the importance
of reproducibility, which is a fundamental aspect of
the scientific method. One key component of reproducibility is sharing and publishing (with DOIs) all
of your data and your code together with execution
details (such as parameters and key intermediate results). This is important because it helps document the
specific work that was done. It helps other researchers
understand the context of the claims you are making.
It also helps others build on what you did and vice
versa. This is something I have been promoting and
pushing in the scientific-paper-of-the future initiative. 1
This initiative promotes a future in which all relevant
details of a computational experiment are exposed
properly and in a structured way.
A positive trend that we are seeing now is that more
and more people want to share their results in the open
as soon as possible. The cycle of waiting for six months
to see your paper come out is too long. People are publishing their work on ArXiv so that their results can be
out and then they move on to the next idea. There
is an incredible eagerness by young researchers to disseminate their work through public sites. We need to
encourage this and do it for data and software as well,
for every research product in addition to papers.
Editorial Team: Many countries have come up with a
national AI policy. What are your thoughts on government involvement in AI research and development?
Gil: The government can play an incredibly im-
portant role. Through its funding programs it can
sustain basic research. The Internet came out of gov-
ernment investment, and so did technologies like chat
bots and search engines, all of which are pervasive in
our society today. So, we are reaping the benefits of
investments in basic research that were made by the
government decades ago. Sometimes government
funding agencies, perhaps influenced by the inter-
ests of the commercial sector in funding the next
start-up, seem very focused on projects that have im-
mediate applications. I think it’s really important for
the government to retain a focus on long-term prob-
lems and to consider sustained decadal-scale funding
programs for the most challenging AI topics.
Editorial Team: What skills are essential for young
AI researchers to have?
Gil: I teach my students to be very mindful of long-term, deep ideas. Students will often focus on the next
paper, or on the things they can do today to extend
some code they are working on. However, one of the
most important skills for any scientist to have is to
think deeply about a very challenging problem. To do
that, you need to invest the time. You need to make
a commitment to spend time defining and thinking
about that problem. It is much harder to think about
what will be important in 10 years than it is to think
about the next paper or the next line of code. It is very
hard to devote yourself to thinking rather than coding, but it is very important if we want fundamental
advances in AI. For students, this is especially hard,
and as professors, we need to teach it.
This is also why it is important for students to go to
a broad conference like Association for the Advancement of Artificial Intelligence (AAAI), to get a good
view of different research topics in AI, which can
promote deeper thinking about problems. For example, at the AAAI conference, you will find sessions on
machine learning and natural language processing,
others on robotics and human computer interaction,
and others on reasoning and planning. And many of
the papers will combine research on several areas of
AI. You will find both interesting applications and
fundamental research. I also attend specialized con-
ferences, but I find the AAAI conference most inspir-
ing for formulating long-term research problems.
Finally, a problem I find that it is when I am I think-
ing about long-term topics that I am most excited,
work the hardest, and get the most satisfaction at a
Editorial Team: Because deeper ideas take time to
develop, and publications may not come about so
quickly, what are some methods you use to know
that you are making progress against deeper ideas?
Gil: This is how I measure progress: I start a folder
on a far-fetched topic and watch how it grows over
time. Initially the folder is very thin, but I keep adding
to it as I think more about the topic. Six months
later it has expanded with details, and in yet another 6 months it will have expanded further, and
so on. At some point, I will have enough of an idea
to work with a student, and eventually, I’ll understand the problem enough to formulate a 5-year