Baidu AI Team Caught Cheating - Banned For A Year From ImageNet Competition
Written by Mike James   
Thursday, 04 June 2015

The race to build the best neural network seems to have descended into tactics that are not familiar in the academic world. Chinese search engine Baidu has built a world class team that delivers first class results - but now it has been caught cheating.

 

imagenettexctimagenet

ImageNet is the benchmark standard for testing convolutional neural networks and other image recognition techniques. It was success on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), that brought Geoff Hinton's approach to convolutional neural networks to everyone's attention and recently Google's GoogLeNet did even better. The competition to produce the best neural network was intense and in the latest breakthrough Baidu's team claimed to have done even better with an error rate of just 4.58%, beating Microsoft and Google's 4.9% and 4.8% - humans have an error rate of around 5%. 

Baidu made much of the small improvement. For example, it posted an article on its technology blog headlined “Baidu Achieves Top Results on Image Recognition Challenge.”

Baidu is serious about AI and hired Google's AI expert Andrew Ng to head the project and then built a very big super computer called Minwa. This is the machine that ran the huge neural network on the ImageNet data and which produced the result that gave Baidu the crown of the best neural network on the planet. 

However the organizers of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) noticed that:

During the period of 28 November 2014 to 13 May 2015, there were at least 30 accounts used by a team from Baidu to the test server at least 200 times, far exceeding the specified limit of two submissions per week.

This is much more than the permitted two submissions per week. 

The report continues:

To make this clear, by exploiting the ability to test many slightly different solutions on the test server it is possible to

1) select the best out of a set of very similar solutions based on test performance and achieve a small but potentially significant advantage and

2) choose methods for further research and development based directly on the test data instead of using only the training and validation data for such choices.

The Baidu team has been asked not to submit to the test server for the next 12 months. In response the company has said that it was a "mistake". 

What are we to make of this?

It could be that that the extra interaction with the test data wasn't taken advantage of - but the restriction on submission rate was imposed to minimise this possibility. By creating 30 accounts to increase the submission rate, it is clear that it wasn't accidental. 

 imagenetsq

More Information

ImageNet Communication

 

Related Articles

Baidu Hires Andrew Ng 

Neural Networks Beat Humans

Google's Neural Networks See Even Better

 

To be informed about new articles on I Programmer, install the I Programmer Toolbar, subscribe to the RSS feed, follow us on, Twitter, FacebookGoogle+ or Linkedin,  or sign up for our weekly newsletter.

 

Banner


Stack Overflow On Google Cloud
06/03/2024

Stack Overflow and Google Cloud have announced a partnership to deliver new gen AI-powered capabilities to developers through the Stack Overflow platform, Google Cloud Console, and Gemini for Google C [ ... ]



Ibis 8 Adds Streaming
05/03/2024

Ibis 8.0 has been released with stream processing backends. The new release includes Apache Flink as a streaming backend, and RisingWave, a streaming database backend. There's also a new batch backend [ ... ]


More News

 

raspberry pi books

 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Thursday, 04 June 2015 )