Invoke, Freeze And GC

In this case we have to deal with the problem that the callback is being executed by a thread that the RealSense system has created and not the UI thread as is the common practice for a UI event. 

To allow other threads to access a UI component you can use the Dispatcher to schedule a call to a method using the UI thread.

That is, if you write:

image1.Dispatcher.Invoke(method);

then the call to method is executed by the UI thread as soon as it becomes available.

In this form the Invoke method blocks and calling thread only continues after the method as finished execution. You can use the InvokeAsync method if you want a non-blocking call to run the method on the UI thread. However, this brings with it some complications of its own. If the calling thread doesn't wait for the completion of the InvokeAsync then it is possible that the callback will complete and be called again before the InvokeAsync has completed. This would result in an ever-growing backlog of frames to process.

For the moment let's continue with the blocking Invoke which ensures that another frame cannot be processed until this one is completed. Of course, it is important that whatever we do with the frame it is processed as quickly as possible to avoid missed frames. 

So now all we have to do is use the Invoke method to set the Image source:

 image1.Dispatcher.Invoke(
        new Action(() => image1.Source = wbm));

You can define a completely new method to pass to the Invoke method, but it is easier to use an anonymous lambda expression.

If you try this out you will once again discover that things don't work. 

The reason is that wbm, the WriteableBitmap, is created on the new thread and on on the UI thread. WPF is set up to detect attempts to work with objects that weren't created using the UI thread. However, it can work with objects that support IFreezable as long as they are frozen first so that no changes can occur. The WriteableBitmap does support IFreezable so our final attempt at making it all work is:

wbm.Freeze();
image1.Dispatcher.Invoke(
      new Action(() => image1.Source = wbm));

Notice that it is the WriteableBitmap object that is frozen, not the wbm variable that references it.

This does work but there is still an intermittent problem - the program occasionally runs out of memory. 

Why?

If you think about what is happening it becomes fairly obvious why:

wbm = data.ToWritableBitmap(0,
                  image.info.width,
                  image.info.height,
                  96.0, 96.0);
wbm.Freeze();
image1.Dispatcher.Invoke(
       new Action(() => image1.Source = wbm));
image.ReleaseAccess(data);

Each time the callback is used it creates a complete new WriteableBitmap object, freezes it and then sets the Image source to reference it. 

What happens to the old WriteableBitmap object?

The answer is that when you set the Image source to the new WriteableBitmap the old one no longer has any references to it and so become eligible for garbage collection. This should all work, but we are creating a very large WriteableBitmap object perhaps sixty times per second. It is possible that the garbage collector will not be called in time to avoid running out of memory. 

What is the solution?

There are a number of possible ways around this problem. The most obvious is not to create a WriteableBitmap every time you retrieve a sample - use the same one and just change its data. However, this raises the problem of how to do this when you have to freeze its data to pass it to the UI thread. To implement this approach you have to pass the ImageData to the UI thread and get it to perform the update of the WriteableBitmap on the UI thread with no call to freeze being needed. 

A simple and direct solution to the occasional running out of memory problem is to place an explicit call to the garbage collector:

wbm = data.ToWritableBitmap(0,
                 image.info.width,
                 image.info.height,
                 96.0, 96.0);
wbm.Freeze();
image1.Dispatcher.Invoke(
       new Action(() => image1.Source = wbm));
GC.Collect();
image.ReleaseAccess(data);

This works but with the risk that the garbage collection might take a long time and so result in missed frames. In practice that garbage collector doesn't have too much to do and it all works.

Putting this all together gives us the final version of the callback function:

pxcmStatus OnNewSample(int mid,
                    PXCMCapture.Sample sample)
{
 PXCMImage image = sample.color;
 PXCMImage.ImageData data;
 image.AcquireAccess(
               PXCMImage.Access.ACCESS_READ,
   PXCMImage.PixelFormat.PIXEL_FORMAT_RGB32,
                                  out data);
 wbm = data.ToWritableBitmap(0,
                           image.info.width,
                          image.info.height,
                                96.0, 96.0);
 wbm.Freeze();
 image1.Dispatcher.Invoke(
      new Action(() => image1.Source = wbm));
 GC.Collect();
 image.ReleaseAccess(data);
 return  pxcmStatus.PXCM_STATUS_NO_ERROR;
}

Depending on what you plan to do with the data, creating big expensive object like WriteableBitmap every time a frame is received might well not be the best way to work. 

How to stop the streaming?

Simple just add another button and call the SenseManager's Close method. To make this work we need to keep a reference to the SenseManager. 

private void Button_Click_1(object sender,
                            RoutedEventArgs e)
{
 sm.Close();
}

Aligned and unaligned streams

When you enable multiple streams you can either set things up so that the callback is triggered when any single frame from any of the enabled streams is available or only when a complete sample of all of the enabled frames is available. 

If you simply enable streams individually then you get an unaligned stream and you have to test to see which of the frames is available for each call of OnNewSample. 

For example, to stream both the color and depth data add a new Image control named image2:

private void Button_Click(object sender,
                            RoutedEventArgs e)
{
 sm = PXCMSenseManager.CreateInstance();
 sm.EnableStream(PXCMCapture.StreamType.
                      STREAM_TYPE_COLOR, 0, 0);
 sm.EnableStream(PXCMCapture.StreamType.
                      STREAM_TYPE_DEPTH, 0, 0);
 PXCMSenseManager.Handler handler =
                new PXCMSenseManager.Handler();
 handler.onNewSample = OnNewSample;
 sm.Init(handler);
 sm.StreamFrames(false);
}

Then you have to process each frame as it arrives but now you have to test to see which which frame has arrived:

pxcmStatus OnNewSample(int mid, PXCMCapture.Sample sample)
{
 if (sample.color != null)
 {
  PXCMImage image = sample.color;
  PXCMImage.ImageData data;
  image.AcquireAccess(
                 PXCMImage.Access.ACCESS_READ,
     PXCMImage.PixelFormat.PIXEL_FORMAT_RGB32,
                                    out data);
  wbm = data.ToWritableBitmap(0,
                             image.info.width,
                            image.info.height,
                                  96.0, 96.0);
  wbm.Freeze();
  image1.Dispatcher.Invoke(
      new Action(() => image1.Source = wbm));
  image.ReleaseAccess(data);
 }
 if (sample.depth != null)
 {
  PXCMImage image = sample.depth;
  PXCMImage.ImageData data;
  image.AcquireAccess(
                PXCMImage.Access.ACCESS_READ,
    PXCMImage.PixelFormat.PIXEL_FORMAT_RGB32,
                                   out data);
  wbm = data.ToWritableBitmap(0,
                            image.info.width,
                           image.info.height,
                                 96.0, 96.0);
  wbm.Freeze();
  image1.Dispatcher.Invoke(
     new Action(() => image2.Source = wbm));
  image.ReleaseAccess(data);
 }
 GC.Collect();
 return pxcmStatus.PXCM_STATUS_NO_ERROR;
}

If you run this program you will notice that there is a lag before the depth data appears compared to the color data. 

If you want to align the data streams then all you have to do is change the way the you enable them. You have to enable them in one go using the streams object:

Replace the two stream enable commands with:

PXCMVideoModule.DataDesc ddesc =
              new PXCMVideoModule.DataDesc();
ddesc.deviceInfo.streams =
   PXCMCapture.StreamType.STREAM_TYPE_COLOR |
    PXCMCapture.StreamType.STREAM_TYPE_DEPTH;
sm.EnableStreams(ddesc);
PXCMSenseManager.Handler handler =
              new PXCMSenseManager.Handler();

You will discover that now the program runs and the two streams appear in the Image controls at the same time. You don't need to change the the OnNewSample callback, but now the if statements are no longer required as both streams are present in each sample.