cations — but adequacy of the knowledge and reasoning components is specific to AI applications.
Sometimes the desired behavior of
the system is clear, as was the case
when IBM was developing Watson to
win the Jeopardy! game show against
human contestants (Ferrucci et al.
2010). Keep track of the performance
of every revision and consider a policy
(as IBM did with Watson) that rejects
any revisions that do not push performance closer to the goal.
When the goal criteria are not as
clear, make extensive use of regression
tests to ensure that solved cases are
never broken in the future. Sometimes,
even regression tests are too precise as
multiple different outcomes may be
equally good. A good technique in
these situations is to build machinery
to automatically identify any changes
in the system’s output after each code
or knowledge base revision. Knowing
what changed after an update is a valuable first step in identifying if development is on the right track.
Add Safeguards and
AI has been known, on occasion, to
produce odd and unpredictable results
due to complex reasoning systems,
large data sets, and large knowledge
bases. Hence, special care should be
taken to verify data produced by AI
subsystems. In addition, there is a premium on testing carefully for
machine-learning systems that do not
have transparent reasoning processes.
This advice should be heeded more
diligently for builders of AI applications
that make use of human input and
applications that are responsible for
making decisions for users. For that
matter, such applications should provide an opt-out capability that lets the
user complete an action without AI
assistance. An AI system is even stronger when it can explain its decisions
and can help users make sense of the
AI’s assistance and better decide if they
prefer to continue making use of it.
Test with Real Data from Users
or Operating Environment
At i2k Connect, we have learned that
there is a long tail of the kinds of doc-
uments humans (and computers) pro-
duce and that may be fed into our doc-
ument enrichment service. During the
early development effort, we focused
on straightforward cases such as
research articles in PDF form and
Microsoft Word documents made up
mostly of text. However, real data
from real users can be drastically dif-
ferent and highly variable. For exam-
ple, we learned that our system did
not properly handle large text files
produced by computer software (such
as log files or data dumps), and need-
ed extra logic to examine each file
before deciding what kinds of process-
ing would be appropriate. In other
examples, roboticists know that robots
must be tested in the real world and
not just simulations, and developers of
personal assistants, chatbots, search
engines, and other tools know that
humans are an unpredictable source of
a wide range of inputs.
For AI to benefit humankind it must
be deployed; for successful deployment, good AI ideas must be integrated into the human context of actual
use and into the IT context of organizations. In this article, we have tried to
summarize what has been learned
about building, maintaining, and
extending AI applications. We have
boiled it down into a simple checklist
for the developers of today and tomorrow.
Going forward, we can expect the
landscape of AI applications to continue to diversify and expand. The revolutions will continue all around us, in
computers and data, as well as sensing.
So, it follows that apps will continue to get more powerful, more knowledgeable, and cover a broader array of
domains and tasks.
It also follows that apps will be
increasingly data driven, guided by
human knowledge. And they will have
a lot more data available, as the Internet of Things takes off.
Finally, intelligent assistants will be
even more proficient at improving
quality of life. The partnership
between human and machine is going
to be stronger and closer. How will this
improve quality of life? Jobs tend to be
more satisfying when we humans are
20 AI MAGAZINE
able to focus on the real work we set
out to do, not distracted by the low-
level clutter that most people are
forced to deal with today, because
computers aren’t powerful enough, or
because no attempt has yet been made
to automate the jobs people don’t
want to do. Intelligent assistants will
deal with the clutter of low-level tasks,
or tasks that require extended concen-
tration, consistency, scale, and so on.
As an example, we see big opportunities with unstructured data. It will no
longer be lost in the cloud — whether
the corporate cloud or the Internet
cloud. We will have the tools to find it
and unlock its connections. We will
also have the tools to extract the essential information from the cluttered
real-time data streams that overwhelm
As the developers of today and
tomorrow address the new opportunities, the history of IAAI conferences
offers lessons in how to build successful deployed AI applications. We have
attempted to distill these lessons to
increase the chances of future success.
In these concluding remarks, we have
just a few final bits of advice.
It is prudent for AI researchers to pay
attention to what is being learned
through engineering practice —
deployed applications — as was hoped
for from the beginning of IAAI. And it
is prudent for practitioners to take
advantage of opportunities to learn
from research, as was hoped for by
colocating the AAAI and IAAI conferences, and by adding the Emerging
Applications track to the IAAI conference in 1997.
It is also wise to pay attention to
what is happening in the rest of the
computing, data, and sensing world.
Factors external to AI are likely to have
the largest impact on what matters, or
what is possible, or where opportunities lie. The biggest impact on how we
are able to build applications today has
come from revolutions that were not
of our own making. Watch for signals
from the periphery.
And finally, to quote Neil Jacobstein,
“AI expands the range of the possible.”
So keep doing it!
In preparing this article, we have drawn