First frame from captureStream() not sending

Question

We're working on a project where people can be in a chat room with their webcams, and they can grab a snapshot of someone's cam at that moment, do some annotations on top of it, and then share that modified picture as if it was their own webcam (like sharing a whiteboard).

Capturing the webcam stream into a canvas element where it can be edited was relatively easy. Finding the canvas element on our page and doing a .getContext('2d') on it, Used an open library to add editing tools to it. Grabbing a stream from that canvas was done like so:

var canvasToSend = document.querySelector('canvas');
var stream = canvasToSend.captureStream(60);
var room = osTwilioVideoWeb.getConnectedRoom();

var mytrack = null;
room.localParticipant.publishTrack(stream.getTracks()[0]).then((publication) => {
    mytrack = publication.track;
    var videoElement = mytrack.attach();
}); 

This publishes the stream alright, but the first frame will not get sent unless you draw something else on the canvas. Let's say you drew 2 circles and then hit Share, the stream will start but will not be shown on the recipients' side unless you draw a line, or another circle, or anything. It seems like it needs a frame change for it to send data over.

I was able to force this with developer tools by doing something like context.fill();, but when I tried adding this after the publishing function, even in a then()... no luck.

Any ideas on how to force this "refresh" to happen?

Answer 1

So it seems it is expected behavior (and thus would make my FF buggy).

From the specs about the frame request algorithm:

A new frame is requested from the canvas when frameCaptureRequested is true and the canvas is painted.

Let's put some emphasis on the "and the canvas as been painted". This means that we need both these conditions, and while captureStream itself, or its frameRate argument ellapsing or a method like requestFrame would all set the frameCaptureRequested flag to true, we still need the new painting...

The specs even have a note stating

This algorithm results in a captured track not starting until something changes in the canvas.

And Chrome indeed seems to generate an empty CanvasCaptureMediaStreamTrack if the call to captureStream has been made after the canvas has been painted.

const ctx = document.createElement('canvas')
  .getContext('2d');
ctx.fillRect(0,0,20,20);
// let's request a stream from before it gets painted
// (in the same frame)
const stream1 = ctx.canvas.captureStream();
vid1.srcObject = stream1;
// now let's wait that a frame ellapsed
// (rAF fires before next painting, so we need 2 of them)
requestAnimationFrame(()=>
  requestAnimationFrame(()=> {
    const stream2 = ctx.canvas.captureStream();
    vid2.srcObject = stream1;
  })
);

<p>stream initialised in the same frame as the drawings (i.e before paiting).</p>
<video id="vid1" controls autoplay></video>
<p>stream initialised after paiting.</p>
<video id="vid2" controls autoplay></video>

So to workaround this, you should be able to get a stream with a frame by requesting the stream from the same operation as a first drawing on the canvas, like stream1 in above example.

Or, you could redraw the canvas context over itself (assuming it is a 2d context) by calling ctx.drawImage(ctx.canvas,0,0) after having set its globalCompositeOperation to 'copy' to avoid transparency issues.

const ctx = document.createElement('canvas')
  .getContext('2d');
ctx.font = '15px sans-serif';
ctx.fillText('if forced to redraw it should work', 20, 20);
// produce a silent stream again
requestAnimationFrame(() =>
  requestAnimationFrame(() => {
    const stream = ctx.canvas.captureStream();
    forcePainting(stream);
    vid.srcObject = stream;
  })
);
// beware will work only for canvas intialised with a 2D context
function forcePainting(stream) {
  const ctx = (stream.getVideoTracks()[0].canvas ||
      stream.canvas) // FF has it wrong...
    .getContext('2d');
  const gCO = ctx.globalCompositeOperation;
  ctx.globalCompositeOperation = 'copy';
  ctx.drawImage(ctx.canvas, 0, 0);
  ctx.globalCompositeOperation = gCO;
}

<video id="vid" controls autoplay></video>