autonomous systems and also to test out methods to
ensure that the complexity of AI can be condensed
into meaningful chunks for individuals who exist
outside of the AI community. Levels of human-machine trust will be more accurate if transparency is
considered (Lyons 2013). Training based on testing
the intentional, task-based, analytical, teamwork, and
environment aspects of transparency can also be used
to establish predictability based on the system’s
behavior in a series of scenarios using techniques like
IS. The present article concludes with a call on the AI
community (that is, designers) to consider the importance of ensuring that future AI systems be not only
trustworthy, but that they be developed and tested
with the appropriate affordances to promote appropriate trust among users and testers.
Arkin, R. C. 2009. Governing Lethal Behavior in Autonomous
Robots. Boca Raton, FL: CRC Press. doi.org/10.1201/
Arkin, R. C.; Ulam, P.; and Wagner, A. R. 2012. Moral Decision Making in Autonomous Systems: Enforcement, Moral
Emotions, Dignity, Trust, and Deception. Proceedings of the
IEEE 100( 3): 571–589. doi.org/10.1109/JPROC.2011.
Baker, C. V.; Salas, E.; Cannon-Bowers, J. A.; and Spector, P.
1992. The Effects of Interpositional Uncertainty and Workload on Team Coordination Skills and Task Performance.
Paper presented at the annual meeting of the Society for
Industrial Organizational Psychology, Montreal.
Blickensderfer, E. L.; Cannon-Bowers, J. A.; and Salas, E.
1998. Cross Training and Team Performance. In Making Decisions Under Stress: Implications for Individual and Team Training, ed. J. A. Cannon-Bowers and E. Salas, 299–312. Washington, DC: American Psychological Association. doi.
Burke, C. S.; Stagl, K. C,; Salas, E.; Pierce, L.; and Kendall, D.
2006. Understanding Team Adaptation: A Conceptual
Analysis and Model. Journal of Applied Psychology 91( 6):
Cannon-Bowers, J. A., and Salas, E. 1998. Team Performance
and Training in Complex Environments: Recent Findings
from Applied Research. Current Directions in Psychological Science 7( 3): 83–87. doi.org/10.1111/1467-8721.ep10773005
Cannon-Bowers, J. A.; Salas, E.; Blickensderfer, E.; and Bowers, C. A. 1998. The Impact of Cross-Training and Workload
on Team Functioning: A Replication and Extension of Initial
Findings. Human Factors 40( 1): 92–101. doi.org/10.1518/
Cannon-Bowers, J. A.; Tannenbaum, S. I.; Salas, E.; and
Volpe, C. E. 1995. Defining Competencies and Establishing
Team Training Requirements. In Team Effectiveness and Decision Making in Organizations, ed. R. A. Guzzo and E. Salas,
333–380. San Francisco: Jossey-Bass.
Chen, G.; Thomas, B.; and Wallace, J. C. 2005. A Multilevel
Examination of the Relationships Among Training Outcomes, Mediating Regulatory Processes, and Adaptive Performance. Journal of Applied Psychology 90( 5): 827–841.
Chen, J. Y., and Barnes, M. J. 2015. Agent Transparency for
Human-Agent Teaming Effectiveness. In Proceedings of the
Cybernetics (SMC), 1381–1385. Piscataway, NJ: Institute for
Electrical and Electronics Engineers. doi.org/10.1109/
Chen, J. Y. C.; Barnes, M. J.; Harper-Sciarini, M. 2011. Supervisory Control of Multiple Robots: Human-Performance
Issues and User-Interface Design. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews,
41( 4): 435–454. doi.org/10.1109/TSMCC.2010.2056682
Cuevas, H. M.; Fiore, S. M.; Caldwell, B. S.; and Strater, L.
2007. Augmenting Team Cognition In Human-Automation
Teams Performing in Complex Operational Environments.
Aviation, Space, and Environmental Medicine 78( 5)(Section 2):
Dahm, W. J. A. 2010. Technology Horizons: A Vision for Air
Force Science and Technology During 2010–2030, 42. Arlington, VA: United States Air Force Headquarters.
Defense Science Board. 2016. Defense Science Board (DSB)
Summer Study on Autonomy, 36. Washington, DC: Office of
the Under Secretary of Defense for Acquisition, Technology,
Defense Science Board. 2012. Defense Science Board (DSB)
Task Force on the Role of Autonomy in DoD Systems, 100. Washington, DC: Office of the Under Secretary of Defense for
Acquisition, Technology, and Logistics.
Department of Defense. 2015. Autonomy Community of Interest (COI) Test and Evaluation, Verification and Validation
(TEVV) Working Group: Technology Investment Strategy 2015–
2018, 4, 9, 14. Washington, DC: Office of the Assistant Secretary of Defense for Research and Engineering. Link: defen-seinnovationmarketplace.mil/resources/OSD_ATEVV_STRA
Desai, M.; Stubbs, K.; Steinfeld, A.; and Yanco, H. 2009. Creating Trustworthy Robots: Lessons and Inspirations from
Automated Systems. Paper presented at the AISB Convention: New Frontiers in Human-Robot Interaction, 8–9 April,
Dzindolet, M. T.; Peterson, S. A.; Pomranky, R. A.; Pierce, L.
G.; and Beck, H. P. 2003. The Role of Trust in Automation
Reliance. International Journal of Human-Computer Studies
58( 6): 697–718. doi.org/10.1016/S1071-5819(03)00038-7
Fischer, K. 2011. How People Talk with Robots: Designing
Dialogue to Reduce User Uncertainty. AI Magazine 32( 4): 31–
Garland, A., and Lesh, N. 2003. Learning Hierarchical Task
Models by Demonstration. Technical Report, Mitsubishi
Electric Research Laboratory (MERL), USA — (January 2002).
Cambridge, MA: Mitsubishi Electric Research Laboratory.
Geiselman, E. E.; Johnson, C. M.; and Buck, D. R. 2013.
Flight Deck Automation: Invaluable Collaborator or Insidious Enabler? Ergonomics in Design: The Quarterly of Human
Factors Applications 21( 3): 22–26. doi.org/10.1177/
Goetz, J.; Kiesler, S.; and Powers, A. 2003. Matching Robot
Appearance and Behavior to Tasks to Improve Human-Robot
Cooperation. Proceedings of the 2003 IEEE International Workshop on Robot and Human Interactive Communication, 55–60.
Piscataway, NJ: Institute for Electrical and Electronics Engineers. doi.org/10.1109/roman.2003.1251796
Gollwitzer, P. M., and Sheeran, P. 2006. Implementing Intentions and Goal Achievement: A Meta-Analysis of Effects and
Processes. Advances in Experimental Social Psychology 38: 69–