Google has just announced an image compression method that is revolutionary in more ways than one. Basically it relies on an AI algorithm to sharpen parts of the image based on what is already there. 

AI upscaling or super-resolution is an interesting, and almost magical, technique that we have looked at before.

The basic idea is that images are highly redundant and, given a low resolution image, a human can perform a better upscaling than simply using image statistics. By knowing that the fuzzy blob is in fact a face, the human can make the features more distinct where most statistical interpolation methods improve the resolution but often blur the image even more. 

You can see that the image on the right has more pixels, but it is blurred by comparison with the blocky low res image. You could probably do better by taking a tool that sharpens the obvious edges. 

If a human can do it then so can AI. Google introduced an AI upscaling program called RAISR back in November 2016. It learns filters that essentially sharpen the image from a set of images and lower resolution images derived from them. 

"With RAISR, we instead use machine learning and train on pairs of images, one low quality, one high, to find filters that, when applied to selectively to each pixel of the low-res image, will recreate details that are of comparable quality to the original. RAISR can be trained in two ways. The first is the "direct" method, where filters are learned directly from low and high-resolution image pairs. The other method involves first applying a computationally cheap upsampler to the low resolution image (as in the figure above) and then learning the filters from the upsampled and high resolution image pairs. While the direct method is computationally faster, the 2nd method allows for non-integer scale factors and better leveraging of hardware-based upsampling."

For example starting from a low resolution image RAISR first upscales the image using a standard technique and then applies a learned filter to each pixel depending on its surroundings. You can see that it works fairly well:

Of course the computational complexity of the per pixel filter is much higher than the standard upscaling but it is well worth it.

While RASIR is useful, at first it seemed only to have limited application  - i.e. to situations where you have a low resolution image that you want to show using a higher resolution display. For example, upscaling standard TV to 4K comes to mind as an obvious application. 

However, Google has now found a devious way to make RASIR a key component in an image compression algorithm. Put simply, what happens is that you send a lower resolution version of the image, hence a smaller file, and then you use RASIR to restore the resolution. The example given is of a 1000x1500 image taking 100Kb. The compression works by only sending a 500x750 reduced quality image i.e. 25Kb and then using RASIR to restore the sharpness to an upscaled version. 

Currently this only works on Google+, but the idea is to spread its use to other apps and services in the future. 

This is a fun idea, but it is only a small step towards what AI-based compression could do for us. AI-based "in painting" and super resolution techniques can intelligently extrapolate what is visible in an image to what should have been visible if the camera was up to the job. By understanding what the image is of, AI can use its intelligence to fill in the missing detail of a face, say. Using this sort of approach you could get much higher compression ratios - but possibly open yourself to the accusation that resulting super resolution image was more a figment of the AI's "imagination" than a reconstructed image.

More Information

Saving you bandwidth through machine learning

Image Optimization

Related Articles

//No Comment - Cardboard Camera, Image Compression With NN & Anemia Detection 

Mozilla Wants To Take A Byte Out Of JPEGs 

Google Zopfli - Nice Compression Shame About The Speed 

Fractal Image Compression 

Google introduces WebP, a new web image format 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info