Neural Network Pioneer Dies

Saturday, 19 March 2011

David Rumelhart, one of the men generally credited with the re-discovery of the back-propagation method of training neural networks, has died, aged 68.

To the AI community, David Rumelhart is best known for presenting, together with Jay McClelland, the algorithm that created the resurgence of interest in neural networks in the early 1980s.

Before this the AI community had abandoned research on neural networks because of the analysis presented by Minsky and Papert that there were lots of very simple things that an artificial neuron couldn't learn. This work was misunderstood at the time to mean that it wasn't worth working on artificial neurons of any kind. Of course, the truth is that it merely indicated that a singly layer of neurons was limited in what it could learn and it said nothing about multiple layers.

Rumelhart

David E. Rumelhart, 1942-2011

At the time a few researchers realised that multiple layers of artificial neurons could learn more, but in the absence of an effective training algorithm it was viewed as ad-hoc and something of a dead end. Rumelhart and McClelland took a more optimistic view and experimented with small simulated neural networks.

A training method that could be used with multi-layer networks had been described as early as 1969, but it was mostly ignored by the AI community still in the grip of Minksy and Papert's dismissal of artificial neurons. It wasn't until the publication of Rumelhart's 1987 paper with Hinton and Williams in Nature that the idea took hold that training general multilayer networks was not only possible but powerful.

Next Rumelhart and McCelland's two-volume book Parallel Distributed Processing (PDP) provided an overall framework in which multilayer neural networks and the backprop algorithm made perfect sense. It also promoted the idea of "distributed coding" - where you seek a coding that isn't efficient in the usual senses but on which makes learning easier.

The paper, the book and the work of the PDP group changed the way that AI was done and gave rise to the machine learning approach based on various developments on the artificial neural network still in use to day. Indeed many of the "big data" analytic methods and even hardware like the Kinect wouldn't be possible without the neural network approach.

Although many AI researchers think of Rumelhart as a computer scientist, he was a psychologist and much of his work was directed towards understanding how humans worked - especially models of cognition, language and perception. Indeed the David E. Rumelhart Prize, a $100,000 award given annually by the Glushko-Samuelson Foundation, is targeted at contributions to “theoretical foundations of human cognition.”

Writing about his colleague and friend Jay McClelland explained:

"He became fascinated by the idea that our minds work at a sub-symbolic level."The idea was that thoughts emerge from neural activity. They're the consequence of the interaction of neurons. He went behind the scenes to look for the actual basis for our thinking ability."

David Rumelhart died on the 13th of March the cause was complications of Pick's disease, an Alzheimer's-like disorder.

Further reading