The race to build the best neural network seems to have descended into tactics that are not familiar in the academic world. Chinese search engine Baidu has built a world class team that delivers first class results - but now it has been caught cheating.

ImageNet is the benchmark standard for testing convolutional neural networks and other image recognition techniques. It was success on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), that brought Geoff Hinton's approach to convolutional neural networks to everyone's attention and recently Google's GoogLeNet did even better. The competition to produce the best neural network was intense and in the latest breakthrough Baidu's team claimed to have done even better with an error rate of just 4.58%, beating Microsoft and Google's 4.9% and 4.8% - humans have an error rate of around 5%. 

Baidu made much of the small improvement. For example, it posted an article on its technology blog headlined “Baidu Achieves Top Results on Image Recognition Challenge.”

Baidu is serious about AI and hired Google's AI expert Andrew Ng to head the project and then built a very big super computer called Minwa. This is the machine that ran the huge neural network on the ImageNet data and which produced the result that gave Baidu the crown of the best neural network on the planet. 

However the organizers of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) noticed that:

During the period of 28 November 2014 to 13 May 2015, there were at least 30 accounts used by a team from Baidu to the test server at least 200 times, far exceeding the specified limit of two submissions per week.

This is much more than the permitted two submissions per week. 

The report continues:

To make this clear, by exploiting the ability to test many slightly different solutions on the test server it is possible to

1) select the best out of a set of very similar solutions based on test performance and achieve a small but potentially significant advantage and

2) choose methods for further research and development based directly on the test data instead of using only the training and validation data for such choices.

The Baidu team has been asked not to submit to the test server for the next 12 months. In response the company has said that it was a "mistake". 

What are we to make of this?

It could be that that the extra interaction with the test data wasn't taken advantage of - but the restriction on submission rate was imposed to minimise this possibility. By creating 30 accounts to increase the submission rate, it is clear that it wasn't accidental. 

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info