Things get a little more complicated when it comes to processing a stream of samples from the RealSense camera. It is quite easy to get it working. but much more difficult to get it working right. We show you how in this second article about RealSense In C#.

Real RealSense In C#

1. Getting Started With RealSense In C#
2. Event Streams
3. A Point Cloud - coming soon
4. Hand Tracking - coming soon

From the first part of this series we know how to get a single sample from the camera, but how do you get a stream of samples?

The answer is fairly obvious as long as you don't dig too deep into the details.

To get a single sample you: 

1. get an instance of the SenseManager:
   sm = PXCMSenseManager.CreateInstance();


2. enable the streams you want data from:
   sm.EnableStream(PXCMCapture.StreamType.
                  STREAM_TYPE_COLOR, 0,0);


3. intialize the processing pipeline:
   sm.Init();
   
   
   
4. call AquireFrame to get wait for a frame to be available:
   pxcmStatus status = sm.AcquireFrame(true);

Once the frame is available you can process the data using QuerySample to retrieve the sample: 

PXCMCapture.Sample sample = sm.QuerySample();

The sample contains one image object for each of the streams you enabled. 

This gives you a single sample. To acquire a stream of samples all you have to do is repeat the AcquireFrame and processing instruction in a loop:

PXCMSenseManager sm=
          PXCMSenseManager.CreateInstance();

sm.EnableStream(PXCMCapture.StreamType.
                 STREAM_TYPE_COLOR,640,480);
sm.Init();
for (;;) {
 sm.AcquireFrame(true);    
 PXCMCapture.Sample sample=sm.QuerySample();
 process the samples   
 sm.ReleaseFrame();
}

When you have finished with the loop you need to dispose of the processing pipeline:

sm.Dispose();

This is all fine and it works, but notice that there is a problem. The tight loop implemented as for(;;) uses the main thread even though the program is just waiting for AcquireFrame to return. 

If you implement things this way using the UI thread then the UI will freeze and your users will be very unhappy. They will be able to see the results of your work but they will not be able to interact with your program at all.

The obvious solution is to take the AcquireFrame, QuerySample, process sample loop and move it onto a separate thread of its own.

This isn't difficult and it is the approach most of the sample programs take. However the API provides a set of methods that will implement the necessary threading for you - why not use it?

Event Based Streaming

Using event based streaming is very easy. You start off with the same steps. 

1. Create an instance of SenseManager.
2. Enable the streams you want to use.

After this things are only slightly different. 

First you have to supply a callback function that will be automatically called every time a particular state occurs. To connect the callback to the state you need to create a instance of the Handler object. This has properties designed to allow you to specify any of a number of callbacks:

• OnConnect - called when the device is connecting or disconnecting
• OnModuleSetProfile - called just before the module configuration is set
• OnModuleProcessedFrame - called when module data is available
• OnNewSample - called when a captured image is available.

If we are streaming raw data from one of the cameras we need to use OnNewSample. When you start to use the more sophisticated processing modules to get gesture or hand data then you need to use OnModuleProcessedFrame.

All you have to do is create a Handler:

PXCMSenseManager.Handler handler =
                new PXCMSenseManager.Handler();

and set the callback property that you want to use. In the case of our example just the OnNewSample callback which is a standard method with the signature:

pxcmStatus Callback(int mid,
                    PXCMCapture.Sample sample)

The mid parameter is a stream identifier if multiple streams are in use - this can be mostly ignored in simple setups. The sample parameter is a Sample object containing the data from the streams that have been enabled.  The Callback returns a status code usually 

pxcmStatus.PXCM_STATUS_NO_ERROR

Once you have the Handler object correctly initialized you can set the processing pipeline read to run using an overloaded version of the init method:

sm.Init(handler);

This specifies the Handler object that contains the references to the callbacks that you want the pipeline to call for each state. 

Finally you can set the pipeline running using the

StreamFrames(bool);

method. If the Boolean parameter is true the call blocks until the streaming is canceled. That is the callback will be repeatedly called but the thread that started the process will hang until the streaming is canceled by calling the Close function. If the Boolean parameter is false then the call returns immediately and the thread that called StreamFrames continues to process instructions and events. The callback is called repeatedly until the Close function is used.

A Simple Video Example

Now that we know the theory let's put it into practice. In turns out that there are a number of problems that arise because of the use of two threads - one for the UI and one for the data processing. 

Start a new WPF project called WpfCallback and add the necessary references and DLL needed to use the Realsense camera - see Getting Started With RealSense In C# if you need detailed instructions.

Next place a button on the form and define its click event handler as: 

private void Button_Click(object sender,
                            RoutedEventArgs e)
{
 PXCMSenseManager sm =
            PXCMSenseManager.CreateInstance();
 sm.EnableStream(PXCMCapture.StreamType.
                     STREAM_TYPE_COLOR, 0, 0);
 PXCMSenseManager.Handler handler =
               new PXCMSenseManager.Handler();
 handler.onNewSample = OnNewSample;
 sm.Init(handler);
 sm.StreamFrames(false);
}

You can see that when the button is click we get a SenseManager, create a Handler and set it's onNewSample to reference a method we have to write called OnNewSample. Then we initialize the pipeline specifying the handler and set the whole thing going with a non-blocking call to StreamFrames. 

Now all we have to do is write the OnNewSample callback and this is where things get a little subtle.

Place an Image control on the form and name it image1 - this is where we will display the video data. 

All we have to do is use the Sample object passed to the callback to extract the video data and then convert it into a WriteableBitmap, as described in the previous part of this series. 

So, a direct approach to the problem would be to first get the image data: 

pxcmStatus OnNewSample(int mid,
                   PXCMCapture.Sample sample)
{
 PXCMImage image = sample.color;
 PXCMImage.ImageData data;

Then convert the ImageData to a WriteableBitmap:

image.AcquireAccess(
      PXCMImage.Access.ACCESS_READ,
      PXCMImage.PixelFormat.PIXEL_FORMAT_RGB32,
      out data);
wbm = data.ToWritableBitmap(0,
                 image.info.width,
                 image.info.height,
                 96.0, 96.0);

Finally we set the WriteableBitmap as the source for the Image control and release the image data:


image1.Source = wbm;
image.ReleaseAccess(data);
return pxcmStatus.PXCM_STATUS_NO_ERROR;
}

If you try this you will discover that you get a runtime error when you try to set the Image source to the WriteableBitmap. The problem is that the UI components are not threadsafe and so they detect any attempt to access them from any thread other than the once that created them. In short, you can access a UI component from the UI thread but no other.