Neural Network pioneer dies
Neural Network pioneer dies
Saturday, 19 March 2011

David Rumelhart, one of the men generally credited with the re-discovery of the back-propagation method of training neural networks, has died, aged 68.

To the AI community, David Rumelhart is best known for presenting, together with Jay McClelland, the algorithm that created the resurgence of interest in neural networks in the early 1980s.

Before this the AI community had abandoned research on neural networks because of the analysis presented by Minsky and Papert that there were lots of very simple things that an artificial neuron couldn't learn. This work was misunderstood at the time to mean that it wasn't worth working on artificial neurons of any kind. Of course the truth is that it merely indicated that a singly layer of neurons was limited in what it could learn and it said nothing about multiple layers. 


David E. Rumelhart, 1942-2011


At the time a few researchers realised that multiple layers of artificial neurons could learn more, but in the absence of an effective training algorithm it was viewed as ad-hoc and something of a dead end. Rumelhart and McClelland took a more optimistic view and experimented with small simulated neural networks.

A training method that could be used with multi-layer networks had been described as early as 1969, but it was mostly ignored by the AI community still in the grip of Minksy and Papert's dismissal of artificial neurons.  It wasn't until the publication of Rumelhart's 1987 paper with Hinton and Williams in Nature that the idea took hold that training general multilayer networks was not only possible but powerful.

Next Rumelhart and McCelland's two-volume book Parallel Distributed Processing (PDP) provided an overall framework in which multilayer neural networks and the backprop algorithm made perfect sense. It also promoted the idea of "distributed coding" - where you seek a coding that isn't efficient in the usual senses but on which makes learning easier.  

The paper, the book and the work of the PDP group changed the way that AI was done and gave rise to the machine learning approach based on various developments on the artificial neural network still in use to day. Indeed many of the "big data" analytic methods and even hardware like the Kinect wouldn't be possible without the neural network approach.

Although many AI researchers think of Rumelhart as a computer scientist, he was a psychologist and much of his work was directed towards understanding how humans worked - especially models of cognition, language and perception. Indeed the David E. Rumelhart Prize, a $100,000 award given annually by the Glushko-Samuelson Foundation, is targeted at contributions to “theoretical foundations of human cognition.”

Writing about his colleague and friend Jay  McClelland explained:

"He became fascinated by the idea that our minds work at a sub-symbolic level."The idea was that thoughts emerge from neural activity. They're the consequence of the interaction of neurons. He went behind the scenes to look for the actual basis for our thinking ability."

David Rumelhart died on the 13th of March the cause was complications of Pick's disease, an Alzheimer's-like disorder. 

Further reading

Marvin Minsky

Neural networks



Computer Languages by Committee - the 1960s

The 1960 saw the growth of interest in computer languages but, unlike today where successful languages are often designed by single-minded enthusiasts, this was the decade of the committee - language  [ ... ]

Thomas J Watson Jr and the IBM 360 Fifty Years On

The first IBM System 360 was launched on 7th April 1964 and with it a new era of computing started - modular, upgradable and eventually a commodity item. The 360 architecture influences everything inc [ ... ]

Other Articles




Last Updated ( Saturday, 19 March 2011 )

RSS feed of news items only
I Programmer News
Copyright © 2017 All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.