This is an interesting story because it suggests that machine learning and neural networks in particular have graduated to being useful in an almost casual sort of way. No longer is the use of a neural network a research project. For want of a better term, it's a maker project.

Jacques Mattheij confesses an early addiction to LEGO and it was only his encounter with computers that saved him. However, he couldn't help but notice that there might be fun and profit to be made from buying it in bulk and repackaging it into sets of identical parts. A slight error with eBay bidding, we have all been there, resulted in the 2 metric tons of LEGO mentioned in the title. 

"And so, the adventure started. In the middle of picking up the lots of lego my van got stolen so we had to make do with an elderly espace, one lot was so large it took 3 trips to pick it all up. By the time it was done a regular garage was stacked top-to-bottom with crates and boxes of lego. "

The intention had been to spend a few pleasant evenings sorting the LEGO by hand but, due to the Ebay error, it had turned into a lifetime challenge. 

The proposed solution was to use technology. 

The problem seems to split into two parts: building a mechanism that could handle and video the LEGO parts and then creating the AI to do the sorting. The write up of the problems of machine handling of LEGO parts is fun but what you need to know is:

Feeding LEGO reliably from a hopper is surprisingly hard.

Needless to say getting good photograph of LEGO parts is also difficult but let us move on to the AI part of the story.

There are apparently more than 38000 shapes and more than 100 colors of LEGO. I knew there were a lot but surely not that many! The selection of the classification method was difficult, but eventually a neural net was selected because handcrafted features and classifiers are much, much harder than anyone ever imagines. What the human eye sees and hence what the human thinks will make a good discriminator isn't generally what is present in the data. We are far to good at pattern recognition and end up thinking its easy or obvious.

A standard neural network was selected, VGC16 in TensorFlow, and tweaked with some Lego specifics. Even before the images could be sent to the neural network, there was image processing to be done (using OpenCV) to get complete images of the parts from stitched and cropped images. The neural network was run on a modest, $700, NVIDA GX1080 GPU.

The biggest problem in training the neural network was creating a training set. A very large set of correctly labeled LEGO parts photos was needed and this was the task that might well have derailed the project. A rough estimate revealed that about six months of solid work would be required to create a labeled dataset. Then Jacques had a brainwave:

I realized I didn’t need to make the training set all by myself. The machine takes and labels images. All I need to do is spot the ones where the computer was wrong and relabel the image correctly. As the neural net learns, there are fewer mistakes, and the labeling workload decreases.

This bootstrapping approach deserves to be better known and it can be used to make the impossible task of building a training set much easier. On day one 500 parts were hand labeled, scanned and used at the training set. The next day 2000 parts were submitted to the machine and it got about half wrong and were hand-corrected, creating a set of 2500 labeled parts. At the next round of the bootstrap another 4000 parts were classified and the error rate was only 10% and correcting the 400 parts produced a training set of 20,000, which is beginning to look as if it might do the job. You can see it in action, slightly slowed down, in this video:

There is clearly more to do and one of the biggest problems is that training is taking a few days. This could be solved with more GPUs, but patience is also a virtue. 

What is important about this project is that it not only resulted in something interesting and fun, it also proves that AI is good enough and easy enough to implement for tasks that until recently would have been research projects or required a team of people. You can almost do off-the-shelf neural networks. It isn't quite that easy because you have to master TensorFlow, or whatever framework you decide to use. You also still need the best GPU-based setup you have money to buy, as training is still one of the most computationally expensive tasks you can take on. 

It only leaves one question.

What will the machine do once the 2 metric tons of LEGO have been sorted?

More Information

Sorting 2 Metric Tons of Lego

Related Articles

Google Releases Object Detector Nets For Mobile

Why Deep Networks Are Better

A Billion Neuronal Connections On The Cheap

Lego Launches Boost Programming Set

A Worm's Mind In A Lego Body

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info