Using the Kinect Depth Sensor

Written by Harry Fairhead

Article Index
Using the Kinect Depth Sensor
Displaying the Image
A depth histogram

Page 3 of 3

Histogramming depth

One of the standard and very basic methods of segmenting an image is to use a depth histogram. Imagine that you are looking at a scene with two objects then all of the points in each object will be roughly the same distance from the Kinect. What this means is that if you plot the histogram of the distances you will see two peaks - one corresponding to each object. You can use these peaks to work out threshold values for dividing up the image into different objects. It doesn't always work and you can easily invent situations where it really doesn't work but the point is that local maxima in the depth histogram generally mean objects.

Start a new Windows Forms project. A Windows Forms project is easier to work with because since .NET 4 there is a chart control which makes it easy to draw a histogram.

Place a button, a PictureBox and a Chart control on the forms. To make it all work you will need to add:

using Microsoft.Kinect;
using System.Drawing.Imaging;
using System.Runtime.InteropServices;
using System.Windows.Forms.
     DataVisualization.Charting;

Clicking the button starts the program running in the usual way:

KinectSensor sensor;
private void button1_Click(object sender,
                               EventArgs e)
{
 sensor = KinectSensor.KinectSensors[0];
 sensor.DepthStream.Enable(
   DepthImageFormat.Resolution320x240Fps30);
 sensor.DepthFrameReady += DepthFrameReady;
 sensor.Start();
}

When a frame is ready to be processed we simply count the number of points at each depth.

void DepthFrameReady(object sender, 
             ImageFrameReadyEventArgs e)
{
 PlanarImage PImage = e.ImageFrame.Image;
 int temp = 0;

There are too many possible depths and counting each one would result in a histogram with a lot of bins so we divide the distance by four which means we only need a quarter of the number of bins:

int[] count = new int[0x1FFF / 4 +1];

Next we scan though the data incrementing the count array for each distance measured:

imageFrame.CopyPixelDataTo(pixelData);
int temp = 0;
int[] count = new int[0x1FFF / 4 + 1];
for (int i = 0; i < imageFrame.PixelDataLength; i++)
{
 temp = ((ushort)pixelData[i]) >> 3;
 count[temp >> 2]++;
}

Notice that the right shift by 3 normalizes the data and the shift by 2 divides it by 4 so that it can be plotted in bins.

Next we chart the data:

chart1.Series[0].Points.Clear();
for (int i = 1; i <( 0x1FFF / 4); i++)
{
 chart1.Series[0].Points.Add(count[i]);
}

Yes you really have to add the data points one by one! Finally we have to give the system time to display the chart and generally catchup so we use DoEvents despite all its problems - its a quick and ok solution in this case. Finally we load the image into the PictureBox using the function described earlier:

  Application.DoEvents();
  pictureBox1.Image = DepthToBitmap(imageFrame);
 }
}

Finally to make sure everything shuts down corectly you need to add:

private void Form1_FormClosing(
 object sender, FormClosingEventArgs e)
{
 sensor.Stop();
}

You also have to customize the Chart control a little to make it work. Use the Property window to access the ChartAreas collection and then access the Axes collection to set reasonable max and min for the X and Y axes. If you don't do this the chart will keep rescaling and it will jump around.

Now if you run the program you will see a real-time histogram of the depth scene displayed in the PictureBox.

Distance4

This is a very rough and ready program but it will do for experimenting with. Your next task is to extract the cut off points from the histogram - i.e. at the minima between the local maxima - and use these to threshold the image to show areas that are likely discrete objects. Your next task is to check that these "objects" form areas of the image that are physically close to one another. You can then add your own algorithms for working out what the objects are and how they are moving. This is why raw depth data is useful.

You can download the code for both the Windows Forms version and the WPF version in a single ZIP file from the CodeBin (note you have to register first).

Histogramming depth

Practical Windows Kinect in C#
Chapter List

Further Reading

Histogramming depth

Practical Windows Kinect in C#Chapter List

Further Reading

Practical Windows Kinect in C#
Chapter List