my model of what they ought to know, what defect in
that knowledge would have produced the error that
they produced. It’s an interesting form of, if not mind
reading, at least mind inferring.
The final lesson was the ubiquity of knowledge, task-specific knowledge. Of course, for example, medicine.
Knowledge about debriefing: How do we get the knowledge out of the head of the expert into the program?
Knowledge about tutoring: How do we transfer that
into the students and knowledge about the general
task? Diagnosis as a particular variety of inference.
Everywhere we looked there was more to know, more to
understand, and more to write down in explicit forms.
These matters of rendering implicit knowledge
explicit, of mind inferring, and of knowledge trans-
fers are all of a kind with Davis’ concern for explana-
tion and transparency in artificial intelligence. He
I’ve been interested in these issues for several decades.
The bad news, for me at least, is after all that time …
the idea that AI programs ought to be explainable is
now in wide circulation. Alas, where were you guys 40
years ago? There’s a lot of interest, of course, in get-
ting understandable AI. There’s lots of experiments in
getting deep learning systems to become more trans-
parent. As many of you know, Dave Gunning has a
DARPA program on “explainable AI,” and the overall
focus in looking at AI not as automation working
alone but as having AI work together with people. All
of these things are going to work better with systems
that are explainable and transparent.
So there’s lots of reasons to want this, the most obvious ones are trust and training. Trust is obvious. If
we’ve got autonomous cars or medical diagnosis programs, we want to know we can trust the result. But I
think training is another issue. If the system makes a
mistake, what ought we to do about it? Should we give
it more examples? What kind of examples? Is there
something it clearly doesn’t know? What doesn’t it
know? How do we explain it to the system? So transparency helps with making the system smarter.
One key issue I think is the representation and inference model. In what sense is the representation and
inference model in our programs either similar to or a
model of human reasoning? It seems to me that the
closer the system’s representations and model of reasoning are to human representations and reasoning,
the easier it’s going to be to bridge that gap and make
A kind of counterexample of this is currently the
vision systems, the deep learning vision systems that
are doing a marvelously impressive job of image labeling for example. They’re said to derive their own representations and that’s great, but it’s also a problem
because they’re deriving their own representations. If
you want to ask them why they thought a particular
picture was George Washington, what could they possibly say?
Now the issue is made a little bit worse by the collec-
tion of papers these days that show that deep learning
vision systems can be thrown off completely by some
image perturbations that are virtually invisible to peo-
ple but cause these systems to get the wrong answer
with very high probability. Now the problem is that
we don’t know what they’re doing and why they’re
doing it so when you show the system an image that
looks to us like a flagpole and it says, “That’s a
Labrador, I’m sure of it,” if we asked them why you
thought so, it’s not clear what kind of answer they can
Now there’s been some work in this area of course, and
to the extent that these systems use representations
that are human derived, they’re better off. There’s
some clever techniques being developed for examining local segments of the decision boundary, but even
so, when you start to talk about local segments of a
decision boundary in a multidimensional space and
hyperplanes, I suspect most people’s eyes are going to
glaze over. It’s not my idea of an intuitive explanation.
Now this work is in its very early stages and I certainly hope that we can come up with much better ways
to make these extraordinarily powerful and successful
systems a whole lot more transparent. But I’m still
fundamentally skeptical that views of a complex statistical process are going to do that.
Which brings me to a claim that I will make, and then
probably get left hung out to dry on, but I will claim
that systems ought to have representations that are
familiar, simple, and hierarchical and inference methods that are intuitive to people. The best test, I think,
is simple. Ask a doctor why they came up with a particular diagnosis and listen to the answer and then ask
one of our machine learning data systems why they
came up with that answer and see about the difference. So let me summarize. If AI’s going to be an effective assistant or partner, it’s going to have to be able to
be trained in focused ways and it’s going to have to be
able to divulge its expertise in a way that makes sense
to the user, not just to the machine learning specialist.
For Davis, greater fidelity in the modeling of
human expertise into AI systems should serve both
intelligibility and instrumentality.
And yet, as Davis underscored, intelligibility —
explainable AI — also comes with some instrumental
cost. Asked if the requirement for explanation and
transparency could limit other aspects of perform-
ance in an AI system, Davis answered:
It will happen, and I actually know this from experi-
ence. I have a paper in Machine Learning from last
spring [March 2016] that has to do with a medical
diagnosis program of sorts where we built the best pos-
sible classifier we could in a system that had about a
1,000-dimensional space. Its AUC [area under curve]
was above 0.9 and the humans who were doing this
task have an AUC of about 0.75. It was great except it
was a black box.
So then, working with Cynthia Rudin, who was then
at MIT, we built machine learning models that were
explicitly designed to be more transparent and simpler, and we measured that performance and now it’s
down to about 0.85. So not only do I know that explanation and transparency will cost you something,
we’re able to calibrate what it costs you in at least one
circumstance. So I think there’s no free lunch, but we
need both of those things.