Multimodal sensing and action methods can le-
verage the recent trends in machine learning (ML)
with an emphasis on deep learning (DL) methods.
DL operates with large amounts of data to train a
statistical model that represents the context from the
data collected. A statistical model is one type of con-
textual analysis from which dynamic information
supports cooperative model adaptation for situation
awareness, coordinates with first-principles mathe-
matical models representing physical phenomena
for situation assessment, and uses social models to
provide situation understanding. However, as the
power of DL continues to grow, there is a need to
consider contextual information as augmented data,
contextual constraints selecting relevant data, and
contextual prediction as forecasted data. Some DL
contemporary research endeavors and future trends
(Blasch, Liu et al. 2018) include the following:
Data quality — use more valuable and contex-
tual data before trying to change the model.
Data augmentation — use normal data extension
techniques and unsupervised generative models.
Class sampling — model relevant context
parameters with equivalent numbers of samples
Ensemble support — train separate networks
for classifier combinations to improve accuracy.
Realistic analysis — ensure validation sets and
test sets come from the same distribution.
Scalability — design computing methods that
expand with more data and model complexity.
Human-level performance metrics — use domain
experts and regular users to compare system
The importance of data management for mul-
timodal sensing and action of context-based AI
systems supports data at rest — provide structure
(that is, translations) between data for integra-
tion, analysis, and storage; data in collect —
leverage the power of modeling from which
data are analyzed for information, delivered as
knowledge, and supports prediction of future
data needs; data in transit — develop a data
as a service architecture that incorporates con-
textual information, metadata, and information
registration to support the systems-of-systems
design; data in motion — use feedback control
loops to dynamically adapt to changing pri-
orities, timescales, and mission scenarios; and
data in use — afford context-based human–
machine interactions based on dynamic mission
priorities, users, and situations to balance needs,
recommendations, and availability (table 2).
One example of these data management methods
for analytics occurs in physics-based and human-derived information fusion (information fusion; Blasch
et al. 2014) that coordinates data collections through a
user-defined operating picture in support of situation
analysis (Blasch 2013). Contemporary issues concern
situational reasoning, knowledge management, and
command and control for AI (where the A in AI could
extend to automated, augmented, or autonomous).
Intelligence is the ability to recall, reason, and predict.
To foster context-based AI requires data models (Blasch,
Ravela, and Aved 2018), situational analysis (Snidaro
et al. 2016), and systems cooperation (Peterson and
Paley 2011). A data model is a computing paradigm
that organizes high-dimensional complex information
for indexing and recall such as context-aware computing. To predict future events requires a mathematical model based on a set of parameters that is
built on repeatable explanations using deductive logic
such as context-adaptive control. Finally, reasoning
leverages understanding from a conceptual model
and coordination with other knowledge for context-enhanced information fusion. Although conceptual
models are not well embedded in machines, there are
evolutionary, experimental, and intuition-from-inductive-logic methods (for example, nonmonotonic
logic) that support knowledge management. To explain
situations or events requires various multimodal sensing, scenario modeling, and knowledge reasoning
approaches for coordinated action. The power of context-aware, context-adaptive, and context-enhanced
multimodal approaches, combined with AI and ML,
expands the employment of future systems such as autonomous cars, UAV swarms, and mobile applications.
This article advocates AI multimodal sensing and
action that combines models, measurements, algorithms, and computing. The next section discusses
multimodal information fusion (data at rest) while the
following one highlights contextual reasoning from a
variety of ML techniques (data in collect). We then discusses the dynamic data-driven applications systems
(DDDAS) paradigm for using models (data in transit).
Type of AI Focus Objective
Type I Reactive machines Identify patterns from rules for immediate action
Type II Limited memory Estimate response using signal processing
Type III Theory of mind Form representations about the world and other agents
Type IV Self-awareness Understand self-conscious to interact with prediction
Table 1. Types of AI.