In yet another AI breakthrough, researchers from the University of Nottingham Computer Vision laboratory, have managed to transform 2D facial images to their 3D counterparts in a most productive and efficient way and have made the project available for the rest of us to enjoy. The best part is that it can even work with Art such as historical portraits and pencil sketches.

Scientifically the process of mapping facial pixels to 3D coordinates falls within the realm of 3D face reconstruction and is one of the most difficult problems that Computer Vision and Graphics research is trying to solve.

The current well established approaches, mainly 3D Morphable Model (3DMM), have proved cumbersome due to their demanding requirements in datasets with a variable versions of both the 2D pictures and their 3D model counterparts, as well as in algorithmic optimization and tweaking

This new approach, presented in a paper "Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression" proposes a novel technique which can successfully handle all those challenges. Feeding its Convolutional Neural Network (CNN) with just a single 2D photograph of a face, captured under any pose or angle, transforms it into its 3D counterpart promptly and without much fuss.

This is possible because its CNN architecture uses a volumetric representation of the 3D facial geometry that allows it to progress directly from a 2D facial image to the corresponding 3D volume without going through the intermediate steps that 3DMM requires.Needless to say,  omission of those steps results in much improved performance.

In training the CNN, datasets of 2D pictures together with their related 3D facial model scans were used so as to enable the network by itself to infer their connection.The outcome of this attempt was that the:

CNN was able to learn directly, in an end-to-end fashion, the mapping from image pixels to the full 3D facial structure geometry (including the non-visible facial parts).

Which made the difference.

As proof of concept, the researches set up a website where anyone can upload any kind of image from selfies to images downloaded from the web, to produce 3D Models. When trying it out you'll notice that it doesn't take long to get to the end result and that it also works with low quality lossy .jpg images.

Supplied samples for demonstration purposes include photographs of Alan Turing, Barack Obama, Elijah Wood and Grace Hopper.

All well, but the unexpected part (which I not sure the researchers even intended), is that can  also work with really asymmetric, disproportional and deformed objects the likes of paintings and pencil sketches! Take Henry VIII, King of England from 1509-1547, for example:

(original picture used from flickr/Lisby)

Or George Villiers, Duke of Buckingham (c. 1620)

(original picture used from flickr/Lisby)

Or pencil sketches like:

(original picture used from flickr/R.)

(original picture used from flickr/Yaira)

So what potential use is there for this technology?

Scientifically, facial reconstruction is valued a lot in forensic science by helping to identify individuals, as the pencil sketch example demonstrates; or in archaeology by helping to reconstruct historical figures allowing them tell their story. It is both science and art and typically requires the involvement of both an artist and forensic expert. Such an algorithm could therefore speed up the process or even act as the scientists' artistic sidekick.

Commercial uses include the health sector, telecommunication, gaming, simulation and even Art.

More Information

The 3D selfie is here

3D Face Reconstruction from a Single Image

Related Articles

Google's Teachable Machine - What it really signifies
Find Your 2000-Year-Old Double With Face Recognition
New Facebook Computer Vision Tags
Bitesnap - Applying Deep Learning to Calorie Counting

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info