Microsoft Research has made a breakthrough in the way in which we can interact with the virtual world and created the HoloDesk. It is all down to 3D tracking using the Kinect and some clever computer graphics.

Who can't have envied the crew of the Enterprise as they got to play in a completely synthetic world, courtesy of the HoloDeck. Well, so far the HoloDeck is just science fiction but Microsoft Research has taken an obvious, but still impressive, step towards creating a HoloDeck - the HoloDesk.

What it does is easy to describe. It allows the user to interact with a 3D projection of objects. The user can pick virtual objects up, move them around and generally play with them as if they were real objects. What might not be so clear from the video, is the fact that there is no tactile feedback involved. The user doesn't feel the virtual objects in any way whatsoever. Given the remarkable properties of human perception, however, it is possible that some sort of phantom touch experience might well develop if used over time. This is because when the visual sense tells you you are touching something the other lesser senses get into line to support it.

What is really interesting is that the HoloDesk works in a relatively simple way. A 45 degree sheet of glass acts as a beam splitter, allowing a projector to superimpose a visual in front of the real 3D area that the hands are moving in. You can think of it as a simple heads up display.

The projection is entirely 2D, but a parallax 3D effect is created by tracking the position of the face and more specifically the eyes using, what else, a Kinect. The idea is that by working out where the user is looking the 2D overlay can be made to correspond to what would have been seen by the user if it was a true 3D scene. 

A Kinect is also used to track the hand positions. Once you know where the hands are, it is easy to do the 3D synthesis needed to place objects correctly. Some clever computation allows the movement of the hands to be mapped to the virtual objects and move them. The way in which the hands and objects are represented as collections of point objects that move coherently with the real world objects is the big breakthrough, allowing the whole computation to be performed in realtime.

That's it - it is all down to 3D tracking using the Kinect and some clever computer graphics.

So what could it be used for?

Well the HoloDesk is fairly constrained in the volume of space it operates in but I suppose it could be used to test out designs for new gadgets. However, without tactile feedback, the holo-world it creates lacks an important aspect of reality that means it cannot be used to train people to perform tasks. For example, being able to juggle in the HoloDesk probably doesn't let you juggle in the real world.

The Microsoft Research team suggests that it could be used for collaboration but this is difficult to conceptualize.  You can see that there is some scope for game playing, but realistically this is a step on the way to something bigger. If only they can solve the problem of allowing the user to touch the virtual objects with feedback then the possibilities are endless.

To be informed about new articles on I Programmer, subscribe to the RSS feed, follow us on Twitter or Facebook or sign up for our weekly newsletter.