Getting started with Microsoft Kinect SDK - Depth
Written by Mike James   
Friday, 08 July 2011
Article Index
Getting started with Microsoft Kinect SDK - Depth
Working with Windows Forms
A depth histogram

Histogramming depth

One of the standard and very basic methods of segmenting an image is to use a depth histogram. Imagine that you are looking at a scene with two objects then all of the points in each object will be roughly the same distance from the Kinect. What this means is that if you plot the histogram of the distances you will see two peaks - one corresponding to each object. You can use these peaks to work out threshold values for dividing up the image into different objects. It doesn't always work and you can easily invent situations where it really doesn't work but the point is that local maxima in the depth histogram generally mean objects.

Start a new Windows Forms project. A Windows Forms project is easier to work with because since .NET 4 there is a chart control which makes it easy to draw a histogram.

Place a button, a PictureBox and a Chart control on the forms. To make it all work you will need to add:

using Microsoft.Research.Kinect.Nui;
using Microsoft.Research.Kinect.Audio;
using System.Drawing.Imaging;
using System.Runtime.InteropServices;
using System.Windows.Forms.
DataVisualization.Charting;

Clicking the button starts the program running in the usual way:

Runtime nui;
private void button1_Click(
object sender, EventArgs e)
{
nui = Runtime.Kinects[0];
nui.Initialize(RuntimeOptions.UseDepth);
nui.DepthStream.Open(
ImageStreamType.Depth,
2,
ImageResolution.Resolution320x240,
ImageType.Depth);
nui.DepthFrameReady +=
new EventHandler<ImageFrameReadyEventArgs>
(DepthFrameReady);
}

When a frame is ready to be processed we simply count the number of points at each depth.

void DepthFrameReady(object sender, 
ImageFrameReadyEventArgs e)
{
PlanarImage PImage = e.ImageFrame.Image;
int temp = 0;

There are too many possible depths and counting each one would result in a histogram with a lot of bins so we divide the distance by four which means we only need a quarter of the number of bins:

int[] count = new int[0x1FFF / 4 +1];

Next we scan though the data incrementing the count array and performing the bit shift adjustment on the data to create a reasonable looking display:

for (int i = 0; i < PImage.Bits.Length; i += 2)
{
temp= (PImage.Bits[i+1]<<8 |
 PImage.Bits[i])& 0x1FFF ;
count[temp >> 2]++;
temp <<= 2;
PImage.Bits[i] = (byte) (temp & 0xFF);
PImage.Bits[i + 1] = (byte) (temp >> 8);
}

Next we chart the data:

chart1.Series[0].Points.Clear();
for (int i = 1; i <( 0x1FFF / 4); i++)
{
chart1.Series[0].Points.Add(count[i]);
}

Yes you really have to add the data points one by one! Finally we have to give the system time to display the chart and generally catchup so we use DoEvents despite all its problems - its a quick and ok solution in this case. Finally we load the image into the PictureBox using the function described earlier:

 Application.DoEvents();
pictureBox1.Image = DepthToBitmap(PImage);
}

Finally to make sure everything shuts down corectly you need to add:

private void Form1_FormClosing(
object sender, FormClosingEventArgs e)
{
nui.Uninitialize();
}

You also have to customize the Chart control a little to make it work. Use the Property window to access the ChartAreas collection and then access he Axes collection to set reasonable max and min for the X and Y axes. If you don't do this the chart will keep rescaling and it will jump around.

Now if you run the program you will see a real-time histogram of the depth scene displayed in the PictureBox.

 

Distance4

This is a very rough and ready program but it will do for experimenting with. Your next task is to extract the cut off points from the histogram - i.e. at the minima between the local maxima - and use these to threshold the image to show areas that are likely discrete objects. Your next task is to check that these "objects" form areas of the image that are physically close to one another. You can then add your own algorithms for working out what the objects are and how they are moving. This is why raw depth data is useful.

Other Articles in this Series

  1. Getting started with Microsoft Kinect SDK
  2. Depth (This article)
  3. Player index
  4. Depth and Video space
  5. Skeletons
  6. The Full Skeleton
You can download the code for both the Windows Forms version and the WPF version in a single ZIP file from the CodeBin (note you have to register first).

 

If you would like to be informed about new articles on I Programmer you can either follow us on Twitter or Facebook or you can subscribe to our weekly newsletter.

Further reading:

On Kinect:

All About Kinect

Getting Started with PC Kinect using the open source drivers

Kinect goes self aware - a warning well we think it's funny and something every Kinect programmer should view before proceding to create something they regret!

On WPF Bitmap handling:

BitmapSource: WPF Bitmaps

BitmapImage

WriteableBitmap

Custom BitmapSource

 

<ASIN:0672333457>

<ASIN:1449394620>



Last Updated ( Monday, 06 February 2012 )