Optimizing conversion of OpenCV Mat to BitmapSource for WPF display

Question

Hello Stack Overflow community,

I'm working on a real-time face detection application using OpenCvSharp in C#. The current code captures frames from a video source and processes them to detect faces and draw bounding boxes around them. These processed frames are then displayed in a WPF Image control.

The current implementation feels less optimized. Specifically, the conversion of Mat to BitmapSource for the WPF display might be introducing unnecessary overhead.

Here's the main portion of my code:

Method that calls the updatedisplay method

public async Task Main(int selectedItemIndex, Image imageControl, CancellationToken token)
    {
        Mat frame = new Mat();
        await UpdateDisplay(frame, imageControl);
    }

---

This is how I am converting the frame

    private async Task UpdateDisplay(Mat frame, Image imageControl)
        {
            await System.Windows.Application.Current.Dispatcher.InvokeAsync(() =>
            {
                var bitmap = OpenCvSharp.Extensions.BitmapConverter.ToBitmap(frame);
                var bitmapSource = ConvertToBitmap(bitmap);
                imageControl.Source = bitmapSource;
            });
        }



        public static BitmapSource ConvertToBitmap(System.Drawing.Bitmap bitmap)
        {
            var bitmapData = bitmap.LockBits(
                new System.Drawing.Rectangle(0, 0, bitmap.Width, bitmap.Height),
                System.Drawing.Imaging.ImageLockMode.ReadOnly, bitmap.PixelFormat);

            try
            {
                var bitmapSource = new WriteableBitmap(bitmapData.Width, bitmapData.Height,
                    bitmap.HorizontalResolution, bitmap.VerticalResolution,
                    PixelFormats.Bgr24, null);

                bitmapSource.Lock();
                bitmapSource.WritePixels(new Int32Rect(0, 0, bitmapData.Width, bitmapData.Height),
                    bitmapData.Scan0, bitmapData.Stride * bitmapData.Height, bitmapData.Stride);
                bitmapSource.Unlock();

                return bitmapSource;
            }
            finally
            {
                bitmap.UnlockBits(bitmapData);
            }
        }

A few things I would like assistance with:

1. Is there a more optimized way to convert the Mat object toBitmapSource for WPF display without using System.Drawing.Bitmap as an intermediary?
2. Any other performance improvements or best practices you recommend for the overall flow and design of the application?
3. I'm also keen on understanding any pitfalls or potential memory leaks with the current approach and how to address them.

Thanks in advance for your insights and recommendations!

Answer 1

My suggestion would be to use a WriteableBitmap, and just copy the data over from your matrix. This should avoid any unnecessary allocations and data copying

Start by creating writeable bitmap with the same size and type of pixels as your matrix. If the size or pixel format can change you might want to check for this before updating.

Then just call

myWriteableBitmap.WritePixels(
    new Int32Rect(0, 0, image.Width, image.Height)
    frame.Data, // pointer to buffer
    frame.AllocatedMemorySize, // buffer size
    frame.Width * bytesPerPixel // stride of source, i.e. number of bytes used for each row
); 

The number of bytes per pixel could either be constant if you only handle one type of images, or a lookup table, or provided by some existing method.