Augmented reality comes in many forms and Microsoft Research seems to have found a new twist on the idea - augmented handheld projectors. The effect is as if the projector was a flashlight illuminating the virtual world.

There are two key ideas used in the system. The first is that, by knowing where the projector is pointing, the system can display the appropriate image from the 3D model. The second is that, by picking up where the user's hands are, the model can be modified to give the impression of an interaction.

The research team has come up with three systems:

1. A hand held pico projector with an infrared camera and an Inertia Measurement Unit (IMU). The IMU is used to work out where the pico projector is pointing and the infrared camera pics up shadow images that can be used to interact with what is projected. The result in this case is a "shadow" based UI. Yes, the user interacts with the virtual world via a shadow puppet play - the temptation to make a rabbit must be overwhelming... Shadow gestures such as a pinch and so on seem to work well.
2. A room projector which uses a mobile projector, but this time a set of four Kinects are used to work out a 3D model for the room and to track both the position of the camera and the user. With a 3D model of the room you get more than just a flat shadow play. Now the projector can reveal virtual features in the full 3D space. For example, you can draw graffiti on a wall and the projector really does look like a flashlight picking out the full work of art. As the user is tracked they interact with the virtual object. Perhaps the strangest idea is to use the projector to cast the shadow of an object that is positioned in 3D space in the room. The shadow gives the impression that a real object is being scanned with a flashlight and the user can interact with it by touching the space where it should be. The shadow of the user's hand provides the feedback - very strange.
3. A unit similar to the room projector is packaged as a headset. The Kinect maps the room and the projector once again allows the user to see the virtual world. The big advantage of this arrangement is that the flashlight track the user's head movement. It is more like wearing a flashlight on your head.

Are there any applications for this idea?

Well the key factor here is that pico projectors aren't expensive and you can expect that the user would already have a Kinect. What this means is that applications don't have to be very serious. There are obvious openings for games and amusements but what about serious applications?

To be informed about new articles on I Programmer  follow us on Twitter or Facebook or subscribe to our weekly newsletter.