Pipe video frames from ffmpeg to canvas without loading the entire video into memory

Question

I am working on a project that involves frame manipulation and I decided to choose node canvas API for that. I used to work with OpenCV Python and there was a cv2.VideoCapture class that takes a video as input and prepares to read the frames of the video and we can loop through the frames one at a time without having to load all the frames at once in memory. Now I tried a lot of ways to replicate the same using ffmpeg, i.e. trying to load frames from a video in an ordered, but "on-demand," fashion.

I tried using ffmpeg as a child process to process frames and standout the frames.

const spawnProcess = require('child_process').spawn,
    ffmpeg = spawnProcess('ffmpeg', [
        '-i', 'test.mp4',
        '-vcodec', 'png',
        '-f', 'rawvideo',
        '-s', '1920*1080', // size of one frame
        'pipe:1'
    ]);
ffmpeg.stdout.on('data', (data) => {
    try {
        // console.log(tf.node.decodeImage(data).shape)
        console.log(`${++i} frames read`)
        //context.drawImage(data, 0, 0, width, height)
        
        
    } catch(e) {
        console.log(e)
    } 
})

The value in the console shows something around 4000 + console logs, but the video only had 150 frames, after much investigating and console logging the data, I found that was buffer data, and it's not processing it for each frame. The on-data function returns the buffer data in an unstructured way I want to read frames from a video and process each one at a time in memory, I don't want to hold all the frames at once in memory or in the filesystem.

I also want to pipe the frames in a format that could be rendered on top of a canvas using drawImage

Answer 1

I found that was buffer data, and it's not processing it for each frame.

Correct.

The on-data function returns the buffer data in an unstructured way I want to read frames from a video and process each one at a time in memory

You cannot rely on STDIO for data framing. The system you're on controls how buffering works between applications. Therefore, the amount of data you get, and its timing, should be considered undefined. You must buffer data in your application.

Now, you don't have to buffer very much, and your scheme doesn't have to be very complicated. Just buffer until you have enough bytes equal or greater to that of what you need for a raw frame. Then, do whatever you need to do to that frame, remove it from the buffer, and continue buffering for the next frame.

Since you're using JavaScript, you might consider doing this with the streams API, which make it easier to handle backpressure and what not. (Afterall, your application may actually process slower than FFmpeg, and you may want to slow down that stream while your application runs, so you don't end up running out of memory.)

• TransformStream - Works in browsers as well as Node.js, but requires a little setup to work with Node.js ChildProcess STDIO streams.
• node:stream/Transform - The built-in Node.js streams. Easy to work with, but won't work in a browser, so keep that in mind if you're building a library for others to use.