As if we didn't have enough of a distraction in the form of the Loebner prize, the Turing Test turned into a circus. Now we have AI Xprize for a robot that can give a TED talk that gets a standing ovation...

...oh no, not more chatbots!

Why is it that just about anyone thinks that they can invent a challenge that will make AI develop faster? One of the things that any AI researcher knows is that it is very easy to, if not exactly fake it, make it look better than it is. You can arrange to "borrow" the observers' intelligence by way of their desire to see the human in everything. We talk to many an inanimate object, after all. 

You would think that the Xprize organization would know better after setting up prizes for things that really do advance technology. For example, in 2004 SpaceshipOne was awarded $10 million for being the first private rocket to fly to the edge of space and back twice within a week. Prizes, if they are big enough, can make a difference.

The new AI Xprize is:

...for the development of artificial intelligence (AI) so advanced that it could deliver a compelling TED Talk with no human involvement. 

We all know what TED talks are and know that they vary from the inspiring and the informative to packaged pseudo science. The prize is being "presented" by TED and my guess is that it will result in some sort of event being held to judge the entries. 

At the moment there are no clear details of the rules because they seem to be still in the process of working out what they are. They even want input from you in the form of suggestions.

A rough outline of what they have in mind is already on the website. The idea seems to be that there will be 100 predetermined topics and the AI agent will be given 30 minutes to prepare a 3-minute talk on one of the topics. The agent doesn't have to be a bipedal humanoid robot, but as the audience gets to vote on its performance it certainly has to have an attractive presentation. After the talk it will be asked to answer some questions on the topic. 

As in the case of the Loebner prize, a more traditional Turing test, where AI agents engage in conversations with humans, it is suggested that a smaller prize is awarded annually for the best performance. The big prize would only be awarded for a "spectacular" TED talk. The size of the the "big" prize hasn't been determined yet. 

Just as the Loebner prize has degenerated into a match between unintelligent chatbots, that simply act as intelligence mirrors and reflect the intelligence of the human back at them, so to could the AI Xprize. 

If the agents know the 100 topics before the contest then they could prepare all of them before hand and simply replay the recorded talk. If the topics are kept secret then the agent would have to behave like IBM's Watson and scan the Internet for information on the topic to piece together extracts and summarize the material. There are already examples of fairly dumb AI agents that can create news stories and summaries of articles. Answering questions on the topic is again a Watson-like task, although Siri might be more appropriate here. 

If you put the whole thing together in a robot body, complete with facial expressions, gestures and a whole "emotion" package then you might have something new. Push the envelope a little further and you might have something better than the average TED talk.

However, once you set a task this specific clever programmers can start to exploit its regularities and the intelligence of the audience to fool them into thinking that agent is indeed intelligent. This is what happened in the case of many of the implementations of the Turing test using chatbots. Attempts to understand language and respond with meaning were quickly replaced by automatic language transformations and tricks to make the observer interpret the output as intelligent. 

It seems likely that the same sort of shortcuts could be applied to the task of giving a talk. They are unlikely to produce a "spectacular" TED talk. Instead the competition will become a race to refine appropriate tricks rather than build a truly comprehending system.

I suppose anything is better than nothing, but the AI Xprize could so easily be about something that is less easy to subvert.