Conclusion and Future Work
Fifteen teams went live in the inaugural Alexa Prize,
and customer ratings improved by about 24 percent
over the duration of the competition (May 8 to Nov
13). An analysis of the technical and scientific detail
for each team in relation to their performance in the
competition led to the following findings:
The following components are key for building an
effective socialbot: ( 1) dialogue manager (DM), ( 2) nat-
ural language understanding (NLU) module, ( 3)
knowledge module, ( 4) response generation, ( 5) conversational user experience (CUX) handler, ( 6) ranking and model selection policy module.
Teams that focused on building CUX modules saw significant gains on ratings and duration. CUX is an
essential component, and the teams who focused
most of their efforts on NLU and DM, with less
emphasis on CUX, were not received as top performers by Alexa users.
A robust NLU system supported by strong domain
coverage leads to high coherence. Teams who invest-
Figure 4. Conversation Duration 90th Percentile for Socialbots During the Competition.
5/10 6/10 7/10 8/10 9/10 10/10 11/1012/10 1/10
Median Duration All_Duration
5/1 6/1 7/1 8/1 9/1 10/1 11/1 12/1
Response Error Rate - Weekly Average All_Duration
Figure 5. Response Error Rate for Individual Utterance-Response
Pairs for All Socialbots and Finalists During the Competition.