MediaPipe segmentation mask for Pose Detection Web

36 Views Asked by At

Following the MediaPipe Pose landmark detection guide for the web (that you can find here) I managed to correctly display and overlay the resulting landamarks on a video. But I am struggling to display the segmentation mask (which should be possible according to this demo.

After setting the option outputSegmentationMasks to true, I run the Task using PoseLandmarker.detectedVideo(). The attribute segmentationMasks is available and returns a list of poses (I guess), and result[0].canvas is an instance of OffscreenCanvas.

I was hoping to draw the resulting mask using

// rendering loop
canvasCtxVideo.save();
canvasCtxVideo.clearRect(0, 0, canvasCamera.clientWidth, canvasCamera.height);
canvasCtxVideo.drawImage(
    result.segmentationMasks[0].canvas,
    0,
    0,
    canvasCamera.clientWidth,
    canvasCamera.height
);
canvasCtxVideo.restore();

but nothing is shown on the HTMLCanvas.

How can I achieve this?

UPDATE:

Since result[0].canvas is an OffsetCanvas I started looking into the Canvas API. This article suggests that you can either use a WebWorker, or convert the canvas to a an ÌmageBitmap using the method OffsetBuffer.transferToImageBitmap().

Unfortunately neither solution works, because:

  1. it is not possible to transfer an OffsetCanvas to a WebWorker if it already has a rendering context.
  2. with the following snippet the canvas remains blank:
const maskCanvas = document.getElementById("mask_canvas")
const maskCanvasCtx = maskCanvas.getContext("bitmaprenderer")

const mask = result.segmentationMasks[0]
const image = mask.canvas.transferToImageBitmap()
maskCanvasCtx.transferFromImageBitmap(image)
0

There are 0 best solutions below