The Loebner prize

For Turing his test of intelligence was intended as a talking point rather than anything real. After all what is the point of seeing if a program actually passes the Turing test? Apart from the fact that we are still nowhere near having a program that could make a serious attempt at the Turing test, most programs are produced to do something more specific than simulate a general intelligence. Left to its own devices one day the Turing test might be passed, by accident as it were.

But in 1990 the test was elevated into a goal in its own right by the offering of a $100,000 prize by Dr Hugh Loebner for the first program to pass the open-ended Turing test.

What's the "open-ended Turing test"? Well essentially the real goal is a program that can carry on a reasonable conversation via a terminal and the "open-ended" part of the specification wouls let the judge direct the conversation to any topic they liked. This is clearly a difficult test because it allows the judge to move the conversation to areas as diverse as arithmetic, poetry, world politics, philosophy and the inner emotions of humanity.

For a machine to cope well with this full range was very definitely out of reach when the prize was initiated and for foreseeable future. Had we insisted on sticking to the "open ended" part of the test the Loebner prize would have been a theoretical one with no challengers.

To make it possible to hold the contest at the time, the hurdle had to lowered and the obvious way to do this was to make the Turing test a "restricted-domain" in which the judges were forced to keep to a small number of topics - romantic relationships, Shakespeare's plays, Burgundy wines and whimsical conversation.

The program now only had to convince the judge that it is a human when talking about small range of knowledge, a more feasible task given the available computing power and AI techniques. Still not easy mind you, but a low enough hurdle to tempt programmers to have a go. The catch was that the full $100,000, plus the specially minted solid 18-carat gold Medal was to be awarded to the first program to pass the open-ended Turing test and only a lesser award of $1500 and a bronze medal to the best restricted-domain program each year.

The Loebner prize medal

The first contest

The first Loebner prize contest was held in on November 8th 1991 at The Computer Museum in Boston. There were ten finalists selected from over 100 entries worldwide. Oddly the set up for the test used eight terminals connected to six computers by phone lines. Two of the terminals were driven by humans hidden in the museum. The reason why I regard this setup as odd is that the use of phone lines that went outside the building was slightly anacronistic. Anyway the phone lines caused a delay, of 45 minutes to the start of the contest. Interestingly it seems to have been the phone lines to the humans that were causing the problem!

The judges slaved all day typing their side of the dialog into the programs and at the end of the day some of the judges were fooled by the machines. More to the point some of the judges were fooled by the humans into thinking that they were machines! It has to be made clear here that it is an assumption of the Turing test that the human reference can do what it likes to influence the outcome of the test, but in this case none of the human challengers were doing anything other than responding normally, i.e. as humans!

Another important point was that the judges were not computer experts - and this coupled with another factor made the test far less satisfactory than it could have been. The second factor was that the contestants were allowed to pick their restricted domain of discourse (i.e. what they were going to talk about!) and as long as it was within the expertise of "ordinary people" then it was deemed acceptable. This, in combination with non-expert judges, made one type of program and its restricted domain a dead certainty to win.

Chatterbots

For years there had been programs that could hold a convincing type of conversation that can best be described as "defensive". They worked by picking up key words in the input sentence and by "turning" the sentence around. For example, if the input contained any of the words no, not etc.. then the program responded with "Why are you being negative?" if the input contained "never" it responded with "Really never?" and so on.

Eliza, developed by Joseph Weizenbaum between 1964 and 1966 is the first example of what are now termed "chatterbots", programs designed to simulate an intelligent conversation with one or more human users via auditory or textual methods.

The Eliza program was based on a human mode of interaction typified by a Rogerian therapist trained not to make any creative input to a conversation, only to keep it going so that patients could explore their feelings. Taking to Rogerian therapist is very like talking to a brick wall with an slightly clever echo!

<ASIN:6130043783>