How to convert optical flow to feature matrix in videos

Hello.

I want to classify videos using optical flow.

After using this code, I got this result.

Which data can I save as matrix for classification?

I think that I can use optical flow data, but there are 4 matrix(Vx, Vy, Orientation, Magnitude). How can I deal with these data?

I will save the data of opticalFlow and use dimensionality reduction. However, I don't know which property can be used for this process.

clear all
close all

vidReader = VideoReader('shahar_walk.avi');
opticFlow = opticalFlowFarneback

h = figure;
movegui(h);
hViewPanel = uipanel(h,'Position',[0 0 1 1],'Title','Plot of Optical Flow Vectors');
hPlot = axes(hViewPanel);

while hasFrame(vidReader)
    frameRGB = readFrame(vidReader);
    frameGray = rgb2gray(frameRGB);  
    flow = estimateFlow(opticFlow,frameGray);
    
    imshow(frameRGB)
    hold on
    plot(flow,'DecimationFactor',[5 5],'ScaleFactor',2,'Parent',hPlot);
    hold off
    pause(10^-3)
end

Answer 1

[Raunak Gupta]

Hi,

I am unsure about what you want to classify in the video using the optical flow estimation workflow. As per my understanding the above code estimate the direction and magnitude of the movement of any pixel in two consecutive frames. So essentially you can use the algorithm to tell if any object which consists of many pixels is moving or not.

For this you may use any of the pair (Vx,Vy) or (Orientation,Magnitude) as a reference to check if an object is moving. These two pairs are related as for every pixel in the image,

Orientation = atan(Vy./Vx); 
Magnitude = sqrt(Vx.^2 + Vy.^2); 

So basically, you may cluster the points which have same (Vx,Vy) in a region that might represent an object because for a single object its velocity for every point will remain same.

I would suggest saving any of the pair (Vx,Vy) or (Orientation,Magnitude) for dimensionality reduction as that will bring all the data available from optical flow estimation.

[耶 王]

Hello, how to cluster after getting the optical flow vector of each pixel by dense optical flow method? Thank you very much.