Pixel position is bigger than image dimension or negative using pinhole camera model

Hi there

I am having a problem with a simple code that uses the pinhole camera model. The problem is when I am trying to locate the pixel position in the image plane of a certain point in the scene it gives me weird results such as negative value or a number byend the image dimension. here is my code where x,y and z are the coordinate of the point I am interested in camx,camy and camz are the coordinate of the camera in the scene all in mm.Image resolution has 1440* 960 pixels (width *height), focal length is 40 mm. rotation matrix is identity because the camera is facing the scene as shown in the image . From the coordination above, the point should be right infront of the camera (Y is the distance from the camera to the house, X is right in the middel of the front view of the hous and z is the height of the camera as shown in the image below) and as a result the pixels should be roughly in the middle of the image but the results is [w,h]= [2970.6, 636.2].

I think i am doing something wrong in the translation matrix but i cant know what is ,

any help please

x=1.067621626992524e+03;
y=-3.718505419300001e+03;
z=2848.17;
camx=-1871.14;
camy=-7763.71;
camz=2848.17;

F=40; % in millimetere
Pixelox=720;
Pixeloy=480;
Pixelx=0.025 ;
Pixely=0.025;
K=[F/Pixelx 0 0;0 F/Pixely 0;Pixelox Pixeloy 1];
R= [1 0 0;0 1 0;0 0 1];
t=[ x-camx y-camy z-camz];
P=[R;t]*K;
tem= [x y z 1]*P;
w=tem(1)/tem(3);
h=tem(2)/tem(3);

Answer 1

[Matt J]

Here are a few errors that I can deduce from your description:

1. You say that the point is "right in front of the camera" and that Y represents the distance from the camera to the house. Doesn't that mean that the y coordinate of the point should be positive while (x,z) should both be approximately equal to camx and camz? This is not the case with the data you've shown.
2. If the Y axis points from the camera to the house, then the R matrix should not be identity. The first and second columns of R should point along the w-axis and h-axis of your image respectively. But with R=eye(3), the second column of R is instead pointing along the Y-axis toward the house and not along your h-axis.
3. The camera parameter t should not depend on any particular (x,y,z). I think you meant instead to have,

                            t= [-camx,-camy,-camz].

[caesar]

Dear Matt J

Thanks alot for your reply.

firstly, yeah the axis are just as shown in the colour coding.

1- your point is valid regarding the y coordinate , however if you notice that y is in the positive direction away from camy (-3.7185, -7763.71)so should be ok. Regarding xand z, again you are right, i mensioned the expected ,rough, position of the point for clarity only.You can try same x and z and will get same result (negative or beyond the image dimention).

2- rotation matrix is very confusing for me , from your reply i think i swap the third and second coloumn of R, right? correct me if I am wrong.

3- I translation of a point in world cordinate to camera cordinate should be done via t=[ Xworld-Xcam Yworld-Ycam Zworld-Zcam] , right?

secondly, as the third dimention (z) will disappear in the conversion process, should i consider the using the Y axis in my image as Zaxis in the code depending on the theory of Projective camera?

thanks agin for your reply

[Matt J]

You can try same x and z and will get same result (negative or beyond the image dimention).

If I modify your code as below,

camx=-1871.14;
camy=-7763.71;
camz=2848.17;

F=40; % in millimetere
Pixelox=720;
Pixeloy=480;
Pixelx=0.025 ;
Pixely=0.025;

K=[F/Pixelx 0 0;0 F/Pixely 0;Pixelox Pixeloy 1];
R= [1 0 0;0 0 -1;0 1 0];
C=[camx camy camz];

P=[eye(3);-C]*R*K;


x=camx; z=camz; y=-3.718505419300001e+03;

tem= [x y z 1]*P;
w=tem(1)/tem(3),
h=tem(2)/tem(3),

then I obtain the correct output, a projection directly to the center of the image,

w =

  720.0000


h =

  480.0000

2- rotation matrix is very confusing for me , from your reply i think i swap the third and second coloumn of R, right? correct me if I am wrong.

Notice above that I've aligned the first two columns of R with your 2D image axes, which I've assumed you want pointing rightward (along 3D X-axis) and downward (along -Z axis) like in Matlab Image Processing Toolbox conventions. The 3rd column must be the optical axis, which in this case is the +Y axis.

3- I translation of a point in world cordinate to camera cordinate should be done via t=[ Xworld-Xcam Yworld-Ycam Zworld-Zcam] , right?

Yes, and notice from the way I built P above that the first part of the matrix multiplication does exactly that,

  [x y z 1]*P   =   [x y z 1]*[eye(3);-C]*R*K;
  
                =   [x-camx,y-camy,z-camz]*R*K;

[caesar]

Thank you very much Matt J. That is really helpful especially the reply to my three points.

[caesar]

Dear Matt J,

i encountered new minor isuue, if we run the same code as below the output is (759.5530, 480) wheraes it should be (680.447) because the new x is to the left of the center and as we assumed that the columns of R are aligned with the 2D image axes, so I think the width should be less than 720. does that mean the first colomn of R should be (-1) and not (1)? or i am missing something here?

also, what happened if the camera position is behind the scene? for example, in the same code assume that we swapped CamX with X? should the third column element of R changed to (-1) too.

best regard

x=-1971.14;
y=-3718.5;
z=2848.17;

camx=-1871.14;
camy=-7763.71;
camz=2848.17;
F=40; % in millimetere
Pixelox=720;
Pixeloy=480;
Pixelx=0.025 ;
Pixely=0.025;
K=[F/Pixelx 0 0;0 F/Pixely 0;Pixelox Pixeloy 1];
R= [1 0 0;0 0 -1;0 1 0];
C=[camx camy camz];
P=[eye(3);-C]*R*K;

tem= [x y z 1]*P;
w=tem(1)/tem(3),
h=tem(2)/tem(3),

[Matt J]

it should be (680.447)

You have some choices. You could make this change,

R =

     1     0     0
     0     0     1
     0     1     0

although this gives you a non-right handed set of camera axes. Alternatively, you could set F=-40, thereby treating the imaging plane as though it were behind the camera (personally, I might do the latter).

also, what happened if the camera position is behind the scene? for example, in the same code assume that we swapped CamX with X?

If you don't want to allow F to be negative, then the third column of R must point from the camera center toward the imaging plane. The first two columns, as before, must be aligned with your 2D image axes.

[caesar]

as always, your response is so quick and highly apprecieted. changing the R based on what you suggested didnt work very well especially with H (with F=-40 &f=40). However, changing the first column to (-1) worked ok. when YCam <Y negative F solves everything, so thank you .

Answer 2

[Image Analyst]

Not sure what you mean by "Y is the distance from the camera to the house, X is right in the middel of the front view of the hous and z is the height of the camera as shown in the image below" Isn't x and y the distances from the optic axis? If not, did you define a new frame of reference where the optic axis is not at (0, 0)? (This would complicate things.)

You say "x,y and z are the coordinate of the point I am interested in camx,camy and camz are the coordinate of the camera in the scene". Let's get this straight. Which set is in the scene (in front of the pinhole) and which set is in the "focal plane" which is where the image from the scene is landing?

Also you specify both of those so that pretty much nails everything else down.

You say "F=40; % in millimetere" but for a pinhole camera, there is no focal length. It's pinhole, so everything is in focus everywhere, regardless of how far behind the pinhole it is.

Also what's this talk about a rotation matrix? Since it's pinhole camera, why would there be any rotation? If the scene and focal plane were parallel, there would be no rotation matrix. Is your scene a plane and either your scene or focal plane not normal to the optic axis?

What are you trying to figure out exactly? h? w? R?

[caesar]

Hi Image Analyst

talking about y and x in my question was to explain the axis which is also shown in the small arrows on the side of the image.

I dont know what you mean by setting new frame because i am not very expert in this subject, However, I am trying to follow the theory in this :

https://uk.mathworks.com/help/vision/ug/camera-calibration.html using simulated environment.

so according to that subject, i should have the coordination of the camera (camx,camy and camz), point of interest cooedination (x,y,z), focal lenght of the camera, pixel size in world unite to find [w,h] of the pixel in image plane.

i tried that in the code but didnt work because i am not sure about T (translation matrix) and Matt J pointed to that i had some error in the R matrix.

I hope now things are better for you.

thanks for your reply in advance

Answer 3

[caesar]

Hi Matt J

its been long time since i accepted this answer and everything works very well. Now, I am facing another issue that is how to constaruct the rotation matrix in case the camera is not looking straight that is rotated around any axis ? for instance , at the moment i am looking at 20 degrees around the z-axis.?

i dont know if I am allowed to post a link here but I had a question submitted already related to this issue in general and not about the camera itself . here it is link.

thanks in advance