Transform camera stream and send to others (even from inactive tab / app)

Question

I am developing module for video calling website that modifies facial featues. Essentially, I get video stream (MediaStream) and I return modified stream.

Currently I am using "canvas.captureStream()" to generate updated stream. I use "requestAnimationFrame" to invoke transformation function that draws on canvas and eventually pushes to output MediaStram. To receive frames from incoming stream, I am attaching it to video.

This setup is working fine as long as user is active on page.

But if user switches to another tab or another application, browser optimization kicks in and "requestAnimationFrame" are paused.

I tried other alternatives, to no avail:

1. setTimeout - browser optimization reduces timeout to "1S" for inactive tab
2. Web worker - not working for me as I have GL processing that cannot be done in worker thread.
3. WebRTC stream transform classes - Restricted to Chrome only.

Kindly provide any alternatives that could solve my problem - transforming incoming video stream and generate output stream that other apps can consume. Receive sort of callback on which I can perform processing even in inactive tabs.

I am open to other input/output options as well.

Would be glad to provide additional information if needed.

Answer 1

You're trying to break the browser power consumption model - those limitations are made exactly to prevent such activities.

1. Possible solution is to move timer to Worker and initiate frame capturing from it?
2. You can use Insertable Stream for Chrome/Edge and Canvas for Firefox, it's not trottling background tabs if there is an AudioContext (https://developer.mozilla.org/en-US/docs/Web/API/setTimeout) - yes, no Chrome support and two implementation will be required, but as is for now. In the future, when IS support will be added to other browsers, you can switch to Insertable Streams completely - it's the best way to process tracks before sending.