Chatbot Eugene Goostman, which has been programmed to have the "personality" of as a 13-year old Ukrainian boy, has "passed" the Turing Test by convincing judges 33% of the time that it was the human rather than the computer program.

The event was organized by the University of Reading's School of Systems Engineering in partnership with RoboLaw, an EU-funded organisation examining the regulation of emerging robotic technologies and took place at the Royal Society in London on June 7, 2014, the 60th anniversary of Turing's death.

Professor Kevin Warwick, who also organized the Largest Turing Test Ever at Bletchely Park to mark the 100th anniversary of Turing's birth on June 23, 2012 said in a statement:

 "In the field of Artificial Intelligence there is no more iconic and controversial milestone than the Turing Test, when a computer convinces a sufficient number of interrogators into believing that it is not a machine but rather is a human. It is fitting that such an important landmark has been reached at the Royal Society in London, the home of British Science and the scene of many great advances in human understanding over the centuries. This milestone will go down in history as one of the most exciting."

However before we accept that this is a real breakthrough for AI we perhaps need to ask more questions about whether this is evidence that computers can learn to think or just that computers can learn tricks,

Alan Turing proposed the idea for "The Imitation Game", which we now refer to as the Turing Test to address the question "Can Machines Think?" and the details are in the 1950 paper Computing Machinery and Intelligence published in the journal, Mind.

Turing decreed that a computer program that could convince human judges that they were conversing with another human 30% of the time would "win" his test - and Eugene Goostman, programmed by Vladimir Veselov, who was born in Russia and now lives in the United States, and Ukrainian-born Eugene Demchenko, who now lives in Russia, is the first to exceed this threshold.

Eugene Grossman, dubbed "The Weirdest Creature in the World", also won the first prize in the 2012, Turing 100th Anniversary competition, but on that occasion he only convinced judges 29% of the time.

The format of the two events was the same.

Judges exchange typed messages with unseen others - some humans and others software "chatbot" programs. They see two "simultaneous comparison" responses to the input they type and have to decide which is the human and which is the chatbot.

For the 2014 Turing Test a total of 150 simultaneous comparison tests, each with 5 minutes of "conversation", were conducted and four sessions, each comprising 25 tests, were open to the members of the public. During each round the hidden pairs switch over between tests and judges change over between sessions.

Five chatbots took part and the line up included Cleverbot, which came third in 2012 and former winners of the Lobner Prize, Elbot and Ultra Hal. The judges for the event included the actor Robert Llewellyn, who played robot Kryten in the sci-fi comedy TV series Red Dwarf, and Lord Sharkey, who led the successful campaign for Alan Turing’s posthumous pardon last year.

Is it the robots or the judges that are most on trial here?

For example, a human might attribute typing errors to another human - however a chatbot can be programmed to slip in such errors. A chatbot is less likely than a human to understand errors, however - but did the judges try such ploys to try to distinguish between them?

Certainly Eugene Grossman's creators came up with a good ploy.  Asked in 2012:, why he had given his creation the personality of a 13-year old Ukranian, Veselov explained:

"13 years old is not too old to know everything and not too young to know nothing."

It does seem rather a cheat in the contest of the Turing test to use an awkward teenager as a conversational partner.

If you want to try a conversation with Eugene and see if you find him convincing you can do so online - although when we tried today the site was down due to overload - which is one way to tell a chatbot from a human!

The circus that currently surrounds stagings of the Turing Test, where chatbots compete to see who can make the best smalltalk, is simply bringing the whole idea into disrepute. Mass media headlines like "Turing Test Passed" are simply a nonsense and reveal a deep lack of understanding. Perhaps we shouldn't be too hard, however, as it is the organizers of such events who court the publicity and help the misunderstanding along. 

Has the Turing Test been passed?

No.

Not even close. 

Not even off the launch pad.

These tricks have been around since Eliza, the first chatbot. They may be clever. They may even be useful but AI they are not.

More Information

Turing test success marks milestone in computing history

Computing Machinery and Intelligence, Turing A.M., (1950) Mind Vol 59, pp443-460

Interact with Eugene Goostman

Related Articles

In Celebration Of Alan Turing

Largest Turing Test Ever

Loebner Prize Judges Could Easily Identify Chatbots

Turing's Test, the Loebner Prize and Chatterbots

Cartoon - The Turing Test

To be informed about new articles on I Programmer, install the I Programmer Toolbar, subscribe to the RSS feed, follow us on, Twitter, Facebook, Google+ or Linkedin,  or sign up for our weekly newsletter.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info