Long gone are the days when you could say a photo never lies. The latest work by Google and MIT can produce perfect photos of things taken through glass windows and chain fences. It is another step on the way to pure computational photography where the final photo isn't something you could ever have actually seen.

We have previously reported on the automatic removal of reflections from photos using the slight displacement of the faint double image that results when you take a photo through glass in  Separating Reflection And Image. However, this new method is much more generally applicable. 

When humans look through a window or through an obstructing fence at a scene they often gain an impression of what things look like that aren't spoiled by the obscuring effects. The reason is that humans are able to subtract the "noise" by moving their head. The different viewpoints allow the distant scene to be viewed in its entirety and the brain does a reasonable job of "in-painting" the missing parts. 

This is all based on the different motion due to parallax that objects at different distances are subject to. When you move your head things that are close move more than things that are far away.

This is the principle that the Google and MIT teams use to separate the background scene from the foreground that is spoiling the imagery. 

The algorithm basically computes the optical flow of the imagery, i.e. how it moves, from a sequence of stills. The stills can be taken from a video or just taken as a set of photos as the camera is moved to say take a panorama. The principle may be simple, but the implementation is tour de force of image processing. 

The following video explains some of the details and shows lots of examples of how it all works: 

The removal of the fence is impressive, but so are the recovered images of what was reflected in the window. The technique could clearly be used not just to make better photos but to recover information that was previously hidden in photos. Given that so few frames are required to do the processing it could also be applied retrospectively.

We seem to be heading towards the day when the camera will be capable of taking photographs that simply couldn't have been taken with a non-computational approach. When you take out your camera and point it at something through a fence or a window and simply press the button to remove the imperfections, the result will be a photo that corresponds more closely to your experience of seeing but it will be a lie by any definition. 

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info