designing and implementing their Phase 1 evaluation
experiments, where they will select a test problem or
problems in the challenge problem areas of data analytics or autonomy; apply their new ML techniques
to learn an explainable model for their problems;
evaluate the performance of their learned ML model
(table 1); combine their learned model with their
explanation interface to create their explainable
learning system; conduct experiments in which users
perform specified tasks using the explainable learning
system; and measure explanation effectiveness by
employing IHMC’s model of the explanation process
(figure 8) and explanation effectiveness measurement
categories (table 1).
The evaluations will include the following experimental conditions: ( 1) without explanation: the XAI
system is used to perform a task without providing
explanations to the user; ( 2) with explanation: the
XAI system is used to perform a task and generates
explanations for every recommendation or decision it
makes and every action it takes; ( 3) partial explanation: the XAI system is used to perform a task and
generates only partial or ablated explanations (to assess various explanation features); and ( 4) control: a
baseline state-of-the-art nonexplainable system is used
to perform a task.
Explainable Learning Systems
Table 2 summarizes the TA1 teams’ technical approaches and Phase 1 test problems.
Deeply Explainable AI
The University of California, Berkeley (UCB) team
(including researchers from Boston University, the
University of Amsterdam, and Kitware) is developing
an AI system that is human understandable by virtue
of explicit structural interpretation (Hu et al. 2017),
provides post hoc (Park et al. 2018) and introspective
(Ramanishka et al. 2017) explanations, has predictive
behavior, and allows for appropriate trust (Huang
et al. 2018). The key challenges of deeply explainable AI (DEXAI) are to generate accurate explanations
of model behavior and select those that are most useful
to a user. UCB is addressing the former by creating
implicit or explicit explanation models: they can implicitly present complex latent representations in understandable ways or build explicit structures that are
inherently understandable. These DEXAI models create
a repertoire of possible explanatory actions. Because
these actions are generated without any user model,
they are called reflexive. For the second challenge, UCB
proposes rational explanations that use a model of the
user’s beliefs when deciding which explanatory actions
to select. UCB is also developing an explanation interface
based on these innovations informed by iterative design
UCB is addressing both challenge problem areas.
For autonomy, DEXAI will be demonstrated in vehicle control (using the Berkeley Deep Drive data set
and the CARLA simulator) (Kim and Canny 2017)
and strategy game scenarios (StarCraft II). For data
This is a cat
(p = 0.93)
• Why did you do that?
• Why not something else?
• When do you succeed?
• When do you fail?
• When can I trust you?
• How do I correct an error? User with
Tomorrow • I understand why
• I understand why not
• I know when you’ll succeed
• I know when you’ll fail
• I know when to trust you
• I know why you erred
This is a cat
It has fur,
It has this feature
41 42 45 45
19 16 18 16
5331131 133 131
0.30.7 0.70.10.1 0.10.9 0.1
1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.6
1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.5 0.5
Figure 3. The XAI Concept.