guage learning into games. Especially giving reliable
real-time feedback to players was an extensively discussed question. The feedback to this collaborative
workshop style was very positive. The multitude of
fascinating ideas developed during this session also
highlight the validity of the approach.
The workshop was organized by Markus Krause,
François Bry, and Mihai Georgescu. The papers of the
workshop are part of the AAAI Press Technical Report
Scaling Speech, Language
Understanding, and Dialogue
Crowdsourcing has rapidly grown and expanded
across different scientific areas as one of the most suc-
cessful strategies to scaling businesses and processes
in a rapid and cost effective way. Since 2005 (when
Amazon launched its microtask crowdsourcing plat-
form, Mechanical Turk) the community of re-
searchers in computer science, linguistics, speech
technology, and so on have been changing their da-
ta collection, data labeling/annotation, data analysis,
and user studies paradigms. Through crowdsourcing,
research can scale and allow more robust models, bet-
ter results, more accurate assessments and conclu-
sions, and more sophisticated user studies. This
workshop brought to light some of the most recent
research and findings in the use of the crowdsourcing
strategies for speech, language understanding, and
dialogue tasks and to kick off a discussion on what
works and what doesn’t.
The workshop brought together crowdsourcing researchers and enthusiasts from both industry and
academe. The workshop included three invited talks.
Gina-Anne Levow (University of Washington) gave a
talk on the predictive aspect of crowdsourcing spoken dialog systems evaluation. A talk given by Jeanne
Parson (Microsoft) focused on how to use the crowd
to identify the best human text-to-speech (TTS) voice
talent. Nancy Chang (Google) gave a talk on the
challenges around crowdsourcing semantic data annotation. The workshop also included three presentations of peer-reviewed papers on research on alternatives to Mechanical Turk for crowdsourcing
(Haofeng Zhou, Denys Baskov, Matthew Lease), using the crowd for gathering task-oriented dialog
learning data (Walter S. Lasecki, Ece Kamar, Dan Bo-hus, Eric Horvitz), and the impact of context in
crowdsourced annotation (Elnaz Nouri). The workshop concluded with a panel discussion on the topic
Does crowdsourcing really work?—successful and less
successful stories, challenges, and learnings with Dan
Bikel (Google Research), Michael Tjalve (
Mi-crosoft/University of Washington), Daniela Braga
(VoiceBox Technologies), Ece Kamar (Microsoft Research), Annika Hämäläinen (Microsoft).
The workshop participants presented and dis-
cussed different approaches to make the best use of
crowdsourcing for data quality and validation, for da-
ta collection and annotation, and for extracting se-
mantic information. The discussion highlighted the
need for industry and academia working together on
the future of crowdsourcing and the desire to have an
open-source, free crowdsourcing platform where re-
searchers can build on top of shared human intelli-
gence task (HIT) templates and antispam shared HIT
templates and antispam rules, automation pipelines,
and best practices, for example, by extending an ex-
isting HIT app. Several presentations highlighted the
importance of the design of crowdsourcing tasks and
the strong correlation between the framing of the
task and the outcome of the study. This includes the
importance of creating tasks that are engaging and
personally relevant. Approaches to dealing with the
risk of bias were presented and the value vs. the com-
plexity of open-ended tasks was also discussed.
The workshop was organized by Daniela Braga
(VoiceBox), Michael Tjalve (Microsoft and University
of Washington), Ece Kamar (Microsoft), Gina-Anne
Levow (University of Washington), Daniel Bikel
(Google), Maxine Eskenazi (Carnegie Mellon University), Jon Barker (University of Sheffeid), Nikko Ström
(Amazon), and Christoph Draxler (Ludwig Maximilian University of Munich). The papers of the workshop were published as AAAI Press Technical Report
Tatiana Josephy is vice president of product at CrowdFlower.
Matt Lease is an assistant professor at the University of
Texas at Austin.
Praveen Paritosh is an engineer and researcher at Google.
Markus Krause is a research associate in the Department of
Computer Science at the Leibniz University Hannover, Germany.
Mihai Georgescu is a research associate at the L3S Research
Center at the Leibniz University Hannover, Germany.
Michael Tjalve is a senior program manager at Microsoft
and an affiliate assistant professor at University of Washington.
Daniela Braga is a senior speech scientist at VoiceBox Technologies.