Going to Pieces: Inventor Sorts 2 Million Lego Blocks with AI

Jacques Mattheij didn't expect to buy two tons of Lego bricks.

But that's what happened after an evening of bidding - or rather, overbidding - on bulk lots of used bricks on eBay. His plan was to resell the bricks at a profit. But he won more than expected, and by morning, he owned more than 2 million pieces.

Now he needed to sort them to get the best price. And he wasn't keen on the idea of hand-sorting the colossal collection. 'It would take several life times to get that all organized,' Mattheij said in his blog.

So Mattheij, an inventor living outside Amsterdam, turned the job to over to a computer. He reasoned that GPU-accelerated deep learning and object recognition can identify objects in images and video, and recognize pedestrians, bicycles and other objects in the path of self-driving cars. So why not apply this AI to his Lego pieces?

Monster Mash-Up

Mattheij loved Lego as a kid. So, a couple of years back he took his own kids to Legoland in Denmark. Amid the rides, restaurants and shops, he spotted fervent fans buying up bins of Lego bricks.

That's when he decided to try his hand at the booming cottage industry of Lego reselling. His idea was to turn it on its head by automating the tedious task of sorting. Complete Lego sets and rare parts sell for many times what unsorted bulk lots do, so he stood to profit if he succeeded.

For his Lego sorting machine, Mattheij first built a proof of concept made of (what else?) Lego. He then spent months tinkering until he cobbled together a working sorter - or, in his words, a 'Frankenstein-like mess of the most unlikely bits and pieces of hardware ever to end up in a single machine.'

These include pieces from a home treadmill, a cash register conveyer belt, two freezer motors, an air tank and a camera - all held together with 'tons of super glue.' You can see the sorter in action in the slow-motion video below.

Lego Sorting Machine

Training the automated sorter to separate Lego pieces was no easy task. For starters, Mattheij counts over 38,000 shapes and more than 100 colors and shades. He wrote and rewrote software to classify Lego parts, but nothing he tried was up to the gargantuan job.

'After six months of coding up features, writing tests and scanning parts, I'd had enough,' he said.

Then he decided to try GPU-accelerated deep learning. After beefing up his skills, he trained a neural network using Keras and TensorFlow deep learning frameworks with cuDNN acceleration running on the NVIDIA GeForce GTX 1080 Ti GPU. The setup can identify tens of thousands of possible part shapes and colors.

'Within hours, I surpassed all of the results I'd managed to painfully scrounge together, feature by feature, over the preceding months,' Mattheij said.

The Lego sorting machine now sorts 4,000 pieces an hour with an accuracy rate of 97 percent, but Mattheij thinks he can boost speed without compromising accuracy. So far, he's sorted more than 130,000 pieces.

'It beats sorting by hand, and it just keeps going, hour after hour,' he said.

To learn more, seeMatthiej's blog or read his article inIEEE Spectrum.

The Lego sorting machine at work, seen in slow motion.