ing systems and that, by supporting such control in
an interactive fashion, user attitudes toward the
learner can be greatly enhanced.
Allowing Users to Specify
Preferences on Errors
People sometimes want to refine the decision boundaries of their learners. In particular, for some classifiers it might be critical to detect certain classes correctly, while tolerating errors in other classes (for
example, misclassifying spam as regular email is typically less costly than misclassifying regular email as
spam). However, refining classifier decision boundaries is a complex process even for experts, involving
iterative parameter tweaking, retraining, and evaluation. This is particularly difficult because among
parameters there are often dependencies that lead to
complex mappings between parameter values and
the behavior of the system.
To address these difficulties, Kapoor and colleagues
(2010) created ManiMatrix (figure 7), a tool for people
to specify their preferences on decision boundaries
through interactively manipulating a classifier’s confusion matrix (that is, a breakdown of the correct and
incorrect predictions it made for each class). Given
these preferences, ManiMatrix employs Bayesian decision theory to compute decision boundaries that minimize the expected cost of different types of errors, and
then visualizes the results for further user refinement.
A user study with machine-learning novices demonstrated that participants were able to quickly and effectively modify decision boundaries as desired with the
ManiMatrix system. This case study demonstrates that
nonexperts can directly manipulate a model’s learning objective, a distinctly different form of input than
choosing examples and labeling them.
An ensemble classifier is a classifier that builds its prediction from the predictions of multiple subclassifiers, each of which are functions over the same space
as the ensemble classifier. Such ensembles often outperform all of their subclassifiers and are a staple of
applied machine learning (for example, AdaBoost by
Freund and Schapire ). A common workflow
for creating ensemble classifiers is to experiment with
different features, parameters, and algorithms
through trial and error or hill-climbing through the
model space. Even for machine-learning experts,
however, this approach can be inefficient and lead to
To facilitate the creation of ensemble classifiers,
Talbot and colleagues (2009) developed EnsembleMatrix, a novel tool for helping people interactively build, evaluate, and explore different ensembles (figure 8). EnsembleMatrix visualizes the current
ensemble of individual learners through a confusion
matrix. The user can then experiment with and evaluate different linear combinations of individual
learners by interactively adjusting the weights of all
models through a single two-dimensional interpolation widget (top right in figure 8). EnsembleMatrix’s
novel interface also allows people to make use of
their visual processing capabilities to partition the
confusion matrix according to its illustrated performance, effectively splitting the ensemble into
subensembles that can be further refined as necessary. A user study showed that EnsembleMatrix
enabled people to create ensemble classifiers on par
with the best published ensembles on the same data
set. Furthermore, they managed to do so in a single,
one-hour session. The study involved participants
ranging from machine-learning novices to experts.
Figure 7. ManiMatrix System.
The ManiMatrix system displays the confusion matrix of the classifier and allows the user to directly increase or decrease the different types
of errors using arrows on the matrix cells. ManiMatrix provides feedback to the user by highlighting cells that change value as a result of
the user’s click (red indicates a decrease and green indicates an increase).
350 13 12 350 13 12 350 13 12 358 10 7
5 205 18 5 205 16 5 205 16 6 204 16
11 16 158 11 16 193 11 16 193 15 14 195
Cloudy Sunny Rain Cloudy Sunny Rain Cloudy Sunny Rain Cloudy Sunny