Disney Research has made a breakthrough in implementing the technique of acquiring depth information from a simple camera scan of a scene. The images can also be stitched together to create hi-res panoramas with or without 3D information.
We all try our hand at a panorama shot - just pan the camera around and then use some autostitch software to put the mosaic together. Computational photography has some algorithms that will do the stitching automatically by automatically identifying the same features in each photo and then applying a transformation to take account of the changed point of view. For a perfect panorama you need to rotate the camera around its optical center, i.e. just rotate the camera.
However, if you just rotate the camera about itself you don't get any parallax effects - which is why it makes the stitching together easier. If you want to get 3D information from the sequence of shots you need parallax. This means rotating the camera mounted on an offset arm or just moving the camera along an arc in your outstretched hand. The big problem with this method is that the parallax makes it more difficult to fit the mosaic together, and this is the problem that the research team has been working on.
The good news is that the methods generalize to a range of camera positions and so to different types of panorama. To quote from the paper:
"As demonstrated in the results, our contributions resolve central issues of existing techniques for both stereo- and monoscopic panorama generation, as well as for any multiperspective imaging method based on stitching, like x-slit, pushbroom or general linear cameras."
Now it's time to watch the amazing video of what sorts of things their new technique can achieve:
The method is described in a paper to be presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 25-27, in Portland, Oregon. It is a complex multistep algorithm that has clearly taken a lot of work to get right. Essentially the extra work is attention to detail and using more sophisticated techniques to correct distortions and to work out the camera's position. The first set of steps corrects the images for lens and camera distortions. Next a flow based up-sampling is used to add parts of the image that might otherwise be missing in the panorama. This produces an artifact free stitching of the images together into a mono or stereo panorama.
"We demonstrated a solution for creating high-quality stereo panoramas at megapixel resolution. To this end, we made two main contributions: ﬁrst, we developed specifically tailored methods for correcting the input data. We proposed techniques to correct the camera orientations, remove undesired vertical parallax and to obtain a compact representation. Secondly, we use optical ﬂow to upsample the angular input resolution to generate the optimal number of rays for a given output resolution on the ﬂy, effectively resolving aliasing."
If you want to fine detail then read the paper. There are probably lots of real applications to which you could put this technique, but it still needs more computing power than you would find in a mobile phone.
A more general point is that it demonstrates just how much information there is in a set of images and how this can be used to create not just a panorama but a high resolution stereo panorama. Computational photography has only just got started.
I couldn't resist the headline, but the news is perfectly serious. IoT hardware is becoming more and more like a full desktop computer. Move over Arduino, the Intel Joule might well crush you with its [ ... ]