models. Once we understand which variables are
causally connected and which are only correlated, we
can make successful predictions in novel situations as
long as the causal variables are the same. Recent work
has formalized the conditions under which causal
models are transportable (Pearl and Bareinboim 2011,
Bareinboim and Pearl 2012, Lee and Honavar 2013).
Idea 7: Portfolio Methods
A third approach to making AI systems robust to
model incompleteness is to adopt portfolio (or
ensemble) methods. As Minsky said, “We usually
know several different ways to do something, so that
if one of them fails, there’s always another.” Ensemble methods are applied universally in machine learning when the computational cost can be managed,
and even deep networks benefit from being combined into ensembles (He et al. 2016).
A line of research that relates closely to Minsky’s
point is the work on portfolio methods in satisfiability
solvers. One of the first such systems was SATzilla (Xu
et al. 2008). A key aspect that is exploited by SATzilla
and other SAT solver portfolios is that they can detect
when they have found a solution to a SAT problem.
This is a very powerful form of metaknowledge that is
not available to machine-learning ensembles.
SATzilla was optimized for a benchmarking competition in which a collection of SAT problem
instances is designed, and the system is given at most
1200 seconds to solve each instance. SATzilla has
been tested on several different benchmark collections. Here, I report the results on the HANDMADE
benchmark, which contains 1490 problem instances.
Figure 13 shows the pipeline of SATzilla. Given a
SAT problem instance, SATzilla first applies two SAT
solvers (presolver1 and presolver2) in sequence with
Figure 12. Effectiveness of Anomaly Detection.
ROC curve (blue) is shifted upwards and two the left.
0 0.2 0.4 0.6 0.8 1
False Alarm Rate
(Normal Points Classified as Anomalies)