Gunrock, a team of students from the University of California, Davis has been awarded the 2018 Alexa Prize for Best Socialbot, winning themselves $500,000. The $1 million for holding an engaging 20-minute conversation stays on the table for next year. 

Amazon is investing a lot of money, time and effort into making  Alexa the leading voice assistant. The three finalist teams of this year's Alexa Prize contest, which was the second time the competition had been staged, were presented with a total of $650,000. As winners, Gunrock pocketed $500,000 to share between around a dozen team members. The runner up, Alquist from the Czech Technical University, which was also one of the 2017 winning teams, took home $100,000 the same sum as last year. The third prize of $50,000 again went to Scotland's Heriot-Watt University, but to a new team Alana.

Teams from fewer countries, 15 rather than 22, entered the contest this time around, but Amazon didn't disclose how many entries were received. Eight finalist teams were selected in February, each being awarded a research grant $250,000 plus Alexa-enabled devices, free Amazon Web Services (AWS) to support their development efforts, access to new Alexa Skills Kit (ASK) APIs, and tools, data and support from the Alexa team to create their socialbots and advance the field of conversational artificial intelligence (CAI).

The challenge for the contest is to:

converse coherently and engagingly for 20 minutes with humans on popular topics and news events.

At the semi-finals phase, for a period of 6 weeks, the bots were rated by Alexa customers. Interactions were initiated by saying “Alexa, let’s chat” to any Alexa-powered device and competing socialbots were randomly invoked in response to this utterance. At the end of the interaction, customers were prompted to provide a rating from 1 to 5 stars (5 being the highest) on how they felt about speaking with that socialbot. During the semi-finals, the average customer rating across all socialbots was 3.2, which was, according to Amazon a 10% improvement from 2017. These customer ratings determined two of the teams moving on to the finals, Alquist and Alana, while; Amazon selected Gunrock as the wildcard finalist. 

During the finals period, Alexa customers continued to interact with and provide feedback to the three finalists with over 1.7 million conversations in a span of 18 weeks. More than 10% of customers’ conversations with the socialbots lasted at least 10 minutes. Final judging took place at Amazon's Seattle headquarters over two days in early November and involved three Interactors who held conversations with the socialbots, as well as professionals across academia and industry who served as Judges. 

According to Dilek Hakkani-Tur on the Alexa blog:

Judging a conversational AI competition is difficult because conversation is inherently subjective; there isn’t a clear right or wrong response at each turn in a dialog, nor a precise definition of what makes a conversation “coherent” or “engaging”. To represent the diversity of thought and perspectives we engaged Judges with a variety of backgrounds and expertise to independently evaluate each socialbots ability.

Last year's winning team, Sounding Board from University of Washington achieved an average conversation duration of  10:22 minutes and an average score of 3.17 (out of 5) from the judges. This year Gunrock only achieved an average duration of 9:59  and a score of 3.1, but of course the judges rating were only between this year's three finalists and the goalposts may have moved. 

What is clear is that there's a way to go before the $1 Million for a 20 minute conversation is awarded - to the team's University rather than its members. Meanwhile progress is being made in lots of areas, see Analyzing Alexa for our report on the research papers produced by the 2017 semi finalists.The  2018 Proceedings of the Alexa Prize are already available and interested universities know that there is still another contest to prepare for. This time around the application period will open in late winter/early spring 2019 with teams being announced in June 2019 for a fall start which should allow the competition to align better align with the academic calendar. Amazon is looking forward to seeing the number of entrants grow and even to parting with the coveted $1 Million for a truly chatty bot.

More Information

Announcing the 2018 Alexa Prize Winner: University of California, Davis

2018 Proceedings of the Alexa Prize

Alexa Prize

Alexa Skills Kit

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info