What is the measurement unit for the optical flow velocity components?

Hello! I have a question: I have a matlab based application in which I compute the optical flow and extract the two velocity components Vx and Vy.

The code: img = imread('img_capture'.jpg); img = rgb2gray(img); glob.images(:,:,1) = img; [Vx, Vy] = OpticalFlowMatlab(glob.images, 1, 4); %%images,alpha, iterations

Next I compute the position for the FOE (focus of expansion) and use this FOE to compute the time-to-contact:

ttc = D_foe/sum(Vmag(:));

where D_foe is the distance from a point to the FOE; Vmag is the magnitude of the optical flow.

I would like to know what is the output unit for the velocity components (D_foe is the distance in pixels) and the output unit for the time-to-contact. For example a value for the ttc is 0.0092. Can this be seconds? When I got this value there was no obstacle in front of the camera.

Answer 1

[David Young]

The units depend on what the function OpticalFlowMatlab does. I don't know this function, so I can't be sure. Normally, the units would be pixels per frame (where by "frame" I mean the time between image frames in the sequence).

To compute ttc, you divide the image distance from the foe by the flow magnitude. The distance will be measured in pixels, and if the optic flow was in pixels per frame, the ttc will be in frames.

Your expression for the ttc looks wrong, though. You need to divide the distance between the foe and an image point by the magnitude of the flow at that point, and then maybe average over a set of such points. I don't know how Vmag has been computed, but taking its overall sum is most unlikely to be correct. This may account for the unlikely result of 0.0092.

Incidentally, if you're trying to measure ttc, you might do better by getting an overall estimate of the image dilation, using for example http://www.mathworks.com/matlabcentral/fileexchange/27093-affine-optic-flow.

[Sully]

Thanks for the quick response! I am trying to understand an application that can be found at http://www.mathworks.com/matlabcentral/fileexchange/22713-optical-flow-based-robot-obstacle-avoidance-with-matlab . The application seems to work fine. I've made some modifications to it. If considered a point from the 3D space that has a projection on the image plane that corresponds to the image center. Then I computed the distance from the image center to the focus of expansion. After that I divided that distance with the magnitude of the optical flow (like in the application). But the result does not seem to be right. If you can please take a look at the link that I've sent to you maybe you can tell me what's wrong. Thanks!

[Jiro Doke]

Have you tried contacting the author of the file? I think he would be the best person to answer your question.

[Sully]

He does not answer!

[David Young]

I'm sorry, but I don't have time to delve into the C++ code in the submission. Looking at the description, it doesn't seem that a proper ttc computation is being done - just a comparison of the average flow with a threshold.

[David Young]

Sully, about your comment: the distance from the image centre to the FoE is not useful for computing ttc. As I said above, to get an estimate of ttc you divide the distance of a point from the FoE by the magnitude of the flow at that point (assuming you have pure translational motion). You can then average such estimates if the depth variation isn't too great. But to understand this theory, you need to go back to the literature, rather than looking at code that doesn't quite do what you want!

[Sully]

David, I was making a mistake in the code above. The actual working code is:

x_cim_1 = 160;

y_cim_1 = 120;

D_foe_1 = abs(x_cim_1 - x_foe) + abs(y_cim_1 - y_foe);

ttc_1 = D_foe_1/mean2(Vmag)

x_cim and y_cim are the coordinates for the center of the image; D_foe_1 is the distance of the center from the FOE; and the ttc was computed as the distance divided by a mean value of the magnitude of the optical flow for the entire image. For the computation of the ttc I followed the paper "AN OPTICAL FLOW-BASED SENSING SYSTEM FOR REACTIVE MOBILE

ROBOT NAVIGATION". For the optical flow (Horn - Schunk method) I used the following code:

function [Vx,Vy] = OpticalFlow(images,alpha,iterations)

%// Calculating optical flow of a sequence of images.

%// images : 3D array that contains a sequence of images. size of images is (imageHeight, imageWidth, frame number)

%// alpha

%// iterations.

[height,width,frames]=size(images);

%//initialzation of u and v

Vx = zeros(height,width);

Vy = zeros(height,width);

for k = 1:frames-1

% //initialization of Ex Ey and Et

Ex = zeros(height-1,width-1,frames-1);

Ey = zeros(height-1,width-1,frames-1);

Et = zeros(height-1,width-1,frames-1);

%//calculating Ex Ey and Et in frame k.

for x = 2:width-1

for y = 2:height-1

Ex(y,x,k) = (images(y+1,x+1,k)-images(y+1,x,k)+images(y,x+1,k)...

-images(y,x,k)+images(y+1,x+1,k+1)-images(y+1,x,k+1)...

+images(y,x+1,k+1)-images(y,x,k+1))/4;

Ey(y,x,k) = (images(y,x,k)-images(y+1,x,k)+images(y,x+1,k)...

-images(y+1,x+1,k)+images(y,x,k+1)-images(y+1,x,k+1)...

+images(y,x+1,k+1)-images(y+1,x+1,k+1))/4;

Et(y,x,k) = (images(y+1,x,k+1)-images(y+1,x,k)+images(y,x,k+1)...

-images(y,x,k)+images(y+1,x+1,k+1)-images(y+1,x+1,k)...

+images(y,x+1,k+1)-images(y,x+1,k))/4;

end

end

for nn = 1:iterations

for x = 2:width-1

for y = 2:height-1

Vxbar = (Vx(y-1,x)+Vx(y,x+1)+Vx(y+1,x)+Vx(y,x-1))/6+...

(Vx(y-1,x-1)+Vx(y-1,x+1)+Vx(y+1,x+1)+Vx(y+1,x-1))/12;

Vybar = (Vy(y-1,x)+Vy(y,x+1)+Vy(y+1,x)+Vy(y,x-1))/6+...

(Vy(y-1,x-1)+Vy(y-1,x+1)+Vy(y+1,x+1)+Vy(y+1,x-1))/12;

%// chapter 12 of Horn's paper

temp = (Ex(y,x,k)*Vxbar+Ey(y,x,k)*Vybar+Et(y,x,k))/(alpha^2 + Ex(y,x,k)^2 + Ey(y,x,k)^2);

%// update u and v

Vx(y,x) = Vxbar-Ex(y,x,k)*temp;

Vy(y,x) = Vybar-Ey(y,x,k)*temp;

end

end

end

end

My question in what is the measurement unit for the ttc? I guess that the unit for the Vx and Vy is pixels/frame. I don't know if I'm wright. Please help me! Thanks!

[David Young]

OK, it's better taking the mean than the sum, though this is still not how I'd estimate ttc, and I think it will only work if the FoE is a long way from the image centre. (Think what happens if the FoE is at the image centre: D_foe_1 will be zero, but the ttc should not be zero).

Incidentally, to measure an image distance, it's more accurate to use Pythagoras's formula rather than the "city block" distance you're using.

Anyway, I think that the unit for the ttc will be in frame intervals. I.e. if the ttc is 10 then collision is expected in 10 x (the time between the two images).

If I was trying to get this to work, I'd apply the program to some test data. For example, take a simple, well-textured image and generate a second image by applying a known transform. Then you know what the flow velocities ought to be, and what the ttc ought to be. Then apply your program to these test images, and look at the results at every stage - the flow estimates as well as the final ttc - and check that they are consistent with what you put in. That should allow you to work out what is going on.

[Sully]

I've computed the ttc like this:

x_cim_1 = 160;

y_cim_1 = 120;

D_foe_1 = sqrt((x_cim_1 - x_foe)^2 + abs(y_cim_1 - y_foe)^2);

ttc_1 = D_foe_1/mean2(Vmag)

where x_cim and y_cim are the coordinates for the center of the image; x_foe and y_foe are the coordinates for the focus of expansion; D_foe is the distance of the center of the image from the FOE. The problem is that when I compute the ttc I get at let's say frame 45, when I'm close to the wall, the value 96, which I think represents the frames remaining to impact. This is not wright. The value, in my opinion, should have been 9.6 frames to impact. So where is my mistake?

[David Young]

There are various possibilities, and I can't really be sure what the problem is. To find out, I'd have to run your code on synthetic images. I still suspect your formula: using the proper distance is an improvement, but as I said above I don't think the distance from the centre of the image to the FoE is really relevant. You need to divide the distance between the foe and an image point by the magnitude of the flow at that point, and then maybe average over a set of such points.

[Sully]

Please send me your email at [e-mail] and I'll send you the code so that you can test it. Thanks!

[David Young]

Sorry - I don't have time to offer that level of support.