Slow motion in C++

Question

I want to do slow motion. I've seen an implementation here: https://github.com/vaibhav06891/SlowMotion

I modified the code to generate only one frame.

#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/video/tracking.hpp>
#include <opencv2/opencv.hpp>
#include <iostream>
#include <fstream>
#include <string>
using namespace cv;
using namespace std;


#define CLAMP(x,min,max) (  ((x) < (min)) ? (min) : ( ((x) > (max)) ? (max) : (x) )  )

int main(int argc, char** argv)
{
    Mat frame,prevframe;
    prevframe = imread("img1.png");
    frame = imread("img2.png");

    Mat prevgray, gray;
    Mat fflow,bflow;

    Mat flowf(frame.rows,frame.cols ,CV_8UC3);   // the forward co-ordinates for interpolation
    flowf.setTo(Scalar(255,255,255));
    Mat flowb(frame.rows,frame.cols ,CV_8UC3);   // the backward co-ordinates for interpolation
    flowb.setTo(Scalar(255,255,255));
    Mat final(frame.rows,frame.cols ,CV_8UC3);

    int fx,fy,bx,by;

    cvtColor(prevframe,prevgray,COLOR_BGR2GRAY);  // Convert to gray space for optical flow calculation
    cvtColor(frame, gray, COLOR_BGR2GRAY);
    calcOpticalFlowFarneback(prevgray, gray, fflow, 0.5, 3, 15, 3, 3, 1.2, 0);  // forward optical flow
    calcOpticalFlowFarneback(gray, prevgray, bflow, 0.5, 3, 15, 3, 3, 1.2, 0);   //backward optical flow

    for (int y=0; y<frame.rows; y++) 
    {
        for (int x=0; x<frame.cols; x++) 
        {
            const Point2f fxy = fflow.at<Point2f>(y,x);
            fy = CLAMP(y+fxy.y*0.5,0,frame.rows);
            fx = CLAMP(x+fxy.x*0.5,0,frame.cols);

            flowf.at<Vec3b>(fy,fx) = prevframe.at<Vec3b>(y,x);

            const Point2f bxy = bflow.at<Point2f>(y,x);
            by = CLAMP(y+bxy.y*(1-0.5),0,frame.rows);
            bx = CLAMP(x+bxy.x*(1-0.5),0,frame.cols);
            flowb.at<Vec3b>(by,bx) = frame.at<Vec3b>(y,x);                  
        }
    }                   
    final = flowf*(1-0.5) + flowb*0.5;  //combination of frwd and bckward martrix
    cv::medianBlur(final,final,3);
    imwrite( "output.png",final);
    return 0;
}

But the result is not as expected.

For the images:

The result is :

Does anyone know what is the problem?

Answer 1

The optical flow algorithm won't work for your test images.

The first problem is that your test images have very little difference in neighbour pixel values. That completely black lines and a single color square give no clues to optical flow algorithm where the image areas moved as the algorithm is not able to process the whole image at once and calculates optical flow with a small 15x15 (as you set it in calcOpticalFlowFarneback) pixels window.

The second problem is that your test images differ too much. The distance between positions of brown square is too big. Again Farneback is not able to detect it.

Try the code with some real life video frames or edit your tests to be less monotonous (set some texture to the square, background and rectangle lines) and bring the squares closer to each other on the images (try 2-10 px distance). You can also play with calcOpticalFlowFarneback arguments (read here) to suit your conditions.

You can use this code to save the optical flow you get to an image for debugging:

Mat debugImage = Mat::zeros(fflow.size(), CV_8UC3);
float hsvHue, magnitude;

for (int x = 0; x < fflow.cols; x++)
{
    for (int y = 0; y < fflow.rows; y++)
    {
        auto& item = fflow.at<Vec2f>(y, x);
        magnitude = sqrtf(item[0] * item[0] + item[1] * item[1]);
        hsvHue = atan2f(item[1], item[0]) / static_cast<float>(CV_PI)* 180.f;
        // div 2 to fit 0..255 range
        hsvHue = (hsvHue >= 0. ? hsvHue : (360.f + hsvHue)) / 2.f;
        debugImage.at<Vec3b>(y, x)[0] = static_cast<uchar>(hsvHue);
        debugImage.at<Vec3b>(y, x)[1] = 255;
        debugImage.at<Vec3b>(y, x)[2] = static_cast<uchar>(255.f * magnitude);
    }
}
cvtColor(debugImage, debugImage, CV_HSV2BGR);
imwrite("OpticalFlow.png", debugImage);

Here pixel flow direction will be represented with color (hue), and pixel move distance will be represented with brightness.

Try to use this images I created:

.

Also note that

for (int y = 0; y < frame.rows; y++)
{
    for (int x = 0; x < frame.cols; x++)
    {
        const Point2f fxy = fflow.at<Point2f>(y, x);
        fy = CLAMP(y + fxy.y*0.5, 0, frame.rows);
        fx = CLAMP(x + fxy.x*0.5, 0, frame.cols);

        flowf.at<Vec3b>(fy, fx) = prevframe.at<Vec3b>(y, x);
        ...

code won't color some flowf pixels that have no corresponding target positions they moved to, and optical flow algorithm can produce such situations. I would change it to:

for (int y = 0; y < frame.rows; y++)
{
    for (int x = 0; x < frame.cols; x++)
    {
        const Point2f fxy = fflow.at<Point2f>(y, x);
        fy = CLAMP(y - fxy.y*0.5, 0, frame.rows);
        fx = CLAMP(x - fxy.x*0.5, 0, frame.cols);
        flowf.at<Vec3b>(y, x) = prevframe.at<Vec3b>(fy, fx);

        const Point2f bxy = bflow.at<Point2f>(y, x);
        by = CLAMP(y - bxy.y*(1 - 0.5), 0, frame.rows);
        bx = CLAMP(x - bxy.x*(1 - 0.5), 0, frame.cols);
        flowb.at<Vec3b>(y, x) = frame.at<Vec3b>(by, bx);
    }
}

With this changed code and my tests I get this output: