As long as you know the Flintstones, and if you don't you've been under a (Bed)rock for too long, then "Fred is talking to Barney in the lounge" will cause you to imagine the scene. Now an AI system can visualize it for real by generating the cartoon to fit.

This is more than just another clever trick with neural networks. It is a sign that AI is moving towards larger systems where deep neural networks do different jobs and work together to create the solution. You could call it the second stage of deep neural networks. A team of researchers from the Allen Institute for Artificial Intelligence, and the University of Illinois Urbana-Champaign are hard at work generating the first new episodes of the Flintstones in 50 years - but be warned they are very short.

The task is interesting, not just because it gives you a way to generate cartoons without any skill or even any artists to draw the frames, but because it has a set of different skills to master. The AI not only has to understand the description, it also has to extract the cartoon elements from the data and reassemble them to create a new cartoon. In this case the system breaks down into to three large subsystems - Composition, Retrieval, and Fusion Networks, or CRAFT for short. The system works in terms of complete graphical entities like "Fred" or "Barney".

The alternative aproach is to take an end-to-end learning system and try to generate the pixels individually. In this case the network would learn the pixel arrangment needed to display "Fred" performing different actions.

To train the networks a new dataset, Flintstones, a densely annotated dataset based on The Flintstones animated series, consisting of over 25000 videos, each 75 frames long was created. An example frame:

The system was tested on seen and unseen data and compared to the pixel approach. That is, the system generated cartoon clips that were new from descriptions that were new.

You can see the results in the following video:

So what does this all mean - apart from being appealing to Flintstone fans?

Yes I suppose given time and effort something like CRAFT could be developed into a cartoon generator and throw thousands of animators out of a job, but computer graphics is already chipping away at that job market. There may well be a market for a system that can take a fairly detailed script and turn it into a cartoon or even a realistic video in the future, but this isn't the importance of the work.

What we have with CRAFT is a complex AI system involving multiple networks that takes a step towards imagining a scene from its description. At the moment that scene is a flat 2D cartoonish affair, but just think what it could be used for if it was a full 3D model. How do you solve problems visually? AI will be doing similar things in the future.

More Information

Imagine This! Scripts to Compositions to Videos

Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem and Aniruddha Kembhavi

Related Articles

Deep Mimic - A Virtual Stuntman

Google Uses AI To Make Better Artists

Atlas and the Backflip

The Virtual Evolution Of Walking

Robots Walking 

Want To Animate A Chair? Easy With Kinect

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info