2

i have broken down my problem to plainmath and could really use some help.

Basis: I have an image. In this image I have several UV-XYZ pairs. So i know the 3d position of serveral Pixels.

Given the following equations from 3d to 2d space.

$ Z_{c} \begin{pmatrix} U \\ V \\ 1 \end{pmatrix} = \begin{pmatrix} f & 0 & px \\ 0 & f & py \\ 0 & 0 & 1 \end{pmatrix} \begin{pmatrix} r_{1} & r_{2} & r_{3} & t_{1}\\ r_{4} & r_{5} & r_{6} & t_{2}\\ r_{7} & r_{8} & r_{9} & t_{3} \end{pmatrix} \begin{pmatrix} X \\ Y \\ Z \\ 1 \end{pmatrix} $

f, px, py are known constants. Many $UV \to XYZ$ pairs are known. I want to get $r_{1} - r_{9}, t_{1}, t_{2}$ and $t_{3}$

I can transform to:

$ \begin{pmatrix} f & 0 & px \\ 0 & f & py \\ 0 & 0 & 1 \end{pmatrix} ^{-1} \begin{pmatrix} U \\ V \\ 1 \end{pmatrix} = \frac 1 Z_{c} \begin{pmatrix} r_{1} & r_{2} & r_{3} & t_{1}\\ r_{4} & r_{5} & r_{6} & t_{2}\\ r_{7} & r_{8} & r_{9} & t_{3} \end{pmatrix} \begin{pmatrix} X \\ Y \\ Z \\ 1 \end{pmatrix} $

The $Z_{c}$ is the Z-Coordinate of the point AFTER the multiplication with the unknown transformation matrix.

$ Z_{c} = \begin{pmatrix} r_{7} & r_{8} & r_{9} & t_{3} \end{pmatrix} \begin{pmatrix} X \\ Y \\ Z \\ 1 \end{pmatrix} $ This leads to these equations for U and V: $ U_{k} = \frac {r_{1}X + r_{2}Y + r_{3}Z + t_{1}} {r_{7}X + r_{8}Y + r_{9}Z + t_{3}} V_{k} = \frac {r_{4}X + r_{5}Y + r_{6}Z + t_{2}} {r_{7}X + r_{8}Y + r_{9}Z + t_{3}} $

The question in the end is quite simple. I need to transform this equation into sth., so I can apply 50-100 $UV \to XYZ$ pairs and get the camera position and rotation. Without the $Z_{c}$ it is pretty easy to transform into sth. like this:

$ \begin{pmatrix} U_{1} + V_{2} + 1 \\ U_{4} + V_{5} + 1 \\ \vdots \end{pmatrix} = \begin{pmatrix} X_{1} & Y_{1} & Z_{1} & 1 & X_{2} & Y_{2} & Z_{2} & 1 & X_ {3} & Y_{3} & Z_{3} & 1 \\ X_{4} & Y_{4} & Z_{4} & 1 & X_{5} & Y_{5} & Z_{5} & 1 & X_ {6} & Y_{6} & Z_{6} & 1 \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \end{pmatrix} \begin{pmatrix} r_{1} \\ r_{2} \\ r_{3} \\ t_{1} \\ r_{4} \\ r_{5} \\ r_{6} \\ t_{2} \\ r_{7} \\ r_{8} \\ r_{9} \\ t_{3} \end{pmatrix} $

I hope you get the idea. I dont know if its correct, but it seems like it. So this is a classical overdetermined linear equation (if you add 100 Rows), which I could have solved with a QR-Decomposition (i hope its called like that). But I can't apply this idea to the new Problem with $Z_{c}$. Leaves me clueless.

xeed
  • 123
  • Come on guys. There needs to be a solution for this. This is the very last piece in the assemblyline to complete my application... – xeed Mar 15 '13 at 19:33

1 Answers1

1

Let me do some rearranging and substituting to give you a more concise formula:

Your very first formula with the substitution $$Z_{c} = \begin{pmatrix} r_{7} & r_{8} & r_{9} & t_{3} \end{pmatrix} \begin{pmatrix} X \\ Y \\ Z \\ 1 \end{pmatrix}$$

gives

$$ \begin{pmatrix} U \\ V \\ 1 \end{pmatrix} \begin{pmatrix} r_{7} & r_{8} & r_{9} & t_{3} \end{pmatrix} \begin{pmatrix} X \\ Y \\ Z \\ 1 \end{pmatrix} = \begin{pmatrix} f & 0 & px \\ 0 & f & py \\ 0 & 0 & 1 \end{pmatrix} \begin{pmatrix} r_{1} & r_{2} & r_{3} & t_{1}\\ r_{4} & r_{5} & r_{6} & t_{2}\\ r_{7} & r_{8} & r_{9} & t_{3} \end{pmatrix} \begin{pmatrix} X \\ Y \\ Z \\ 1 \end{pmatrix}$$

and now we have

$$ \left[\begin{pmatrix} f & 0 & px \\ 0 & f & py \\ 0 & 0 & 1 \end{pmatrix} \begin{pmatrix} r_{1} & r_{2} & r_{3} & t_{1}\\ r_{4} & r_{5} & r_{6} & t_{2}\\ r_{7} & r_{8} & r_{9} & t_{3} \end{pmatrix}-\begin{pmatrix}0 & 0 & U \\ 0 & 0 & V \\ 0 & 0 & 1 \end{pmatrix} \begin{pmatrix} r_{1} & r_{2} & r_{3} & t_{1}\\ r_{4} & r_{5} & r_{6} & t_{2}\\ r_{7} & r_{8} & r_{9} & t_{3} \end{pmatrix} \right] \begin{pmatrix} X \\ Y \\ Z \\ 1 \end{pmatrix} = 0$$ or $$ \begin{pmatrix} f & 0 & px - U \\ 0 & f & py - V \\ 0 & 0 & 0 \end{pmatrix} \begin{pmatrix} r_{1} & r_{2} & r_{3} & t_{1}\\ r_{4} & r_{5} & r_{6} & t_{2}\\ r_{7} & r_{8} & r_{9} & t_{3} \end{pmatrix} \begin{pmatrix} X \\ Y \\ Z \\ 1 \end{pmatrix} = 0$$

$$ \begin{pmatrix} f & 0 & px - U \\ 0 & f & py - V \\ 0 & 0 & 0 \end{pmatrix} \begin{pmatrix} r_{1}X + r_{2}Y + r_{3}Z + t_{1}\\ r_{4}X + r_{5}Y + r_{6}Z + t_{2}\\ r_{7}X + r_{8}Y + r_{9}Z + t_{3} \end{pmatrix} = 0$$ This gives the two equations

$$ fXr_{1} + fYr_{2} + fZr_{3} + ft_{1}+ (px - U)Xr_{7} + (px - U)Yr_{8} + (px - U)Zr_9 + (px - U)t_{3} = 0$$ $$ fXr_{4} + fYr_{5} + fZr_{6} + ft_{2}+ (py - V)Xr_{7} + (py - V)Yr_{8} + (py - V)Zr_9 + (py - V)t_{3} = 0$$

I tried to format into matrix times the r and t vector, but the equation was too wide to fit. This formulation allows for the separation into unknowns though.

adam W
  • 5,565
  • Can't edit anymore:

    Thanks man, can't I transform your last 2 equations to this: $ \begin{pmatrix} fX & fY & fZ & f & 0 & 0 & 0 & 0 & (px-U)X & (px-U)Y & (px-U)Z & (px-U) \ 0 & 0 & 0 & 0 & fX & fY & fZ & f & (py-V)X & (py-V)Y & (py-V)Z & (py-V) \ \end{pmatrix} \begin{pmatrix} r_{1} \ r_{2} \ r_{3} \ t_{1} \ r_{4} \ r_{5} \ r_{6} \ t_{2} \ r_{7} \ r_{8} \ r_{9} \ t_{3} \end{pmatrix} = 0 $

    I should be able to solve this with a QR-Decomposition or another method.

    Did you mean this formulation does not fit on the site? Then I'm sorry. ^^

     – xeed Mar 16 '13 at 20:40
  • Yeah, that's the one. On my display it was cut off, I figured you could take it from there. – adam W Mar 16 '13 at 21:20
  • Well, I tried this equation in matlab. Code: http://pastebin.com/iGwkru4S I'm not a Matlab genius, but I tried solving it according to this: http://www.mathworks.de/de/help/matlab/math/systems-of-linear-equations.html The message I get is that the matrix has a rank of 11. I dont really know why. I'll continue to figure it out, but maybe you are able to get me there faster. – – xeed Mar 17 '13 at 14:23
  • The matrix should have rank less than $12$ in order to have a non-zero solution for the r/t vector. If it has rank less than $11$ then there are more than one solution. So it sounds right that it has rank $11$. – adam W Mar 17 '13 at 16:09
  • Sooo, i'm pretty sure that i'm right. There are 12 variables, so the matrix needs to have rank 12 to have only one solution. I finally got some non zero values from this equation using SVD. The good thing: rank 11 leaves only a unsolved factor, which i have to apply to get the correct values for Rt. If i can't figure out how to get a rank 12 koefficientmatrix, i PROBABLY can figure this factor out by applying some rotationmatrix properties. And again. Thanks to you, you were huge help. I'm actually looking forward to go to work tomorrow. ^^ – xeed Mar 18 '13 at 00:57
  • Awesome. Remember that the equation is $Ax = 0$, and if $A$ has full rank, then $x = 0$ is the only solution. If $A$ has lesser rank, then it is the null space of $A$ that gives the interesting non-zero solutions for $x$. The null space can be recovered easily from the Q in QR factorization. Your result will be unique only up to scale. – adam W Mar 18 '13 at 01:07
  • You are pretty fast. ^^ So I used the property of det(R)=1 for any rotation matrix. I scale it by the factor the current det offers me. Works pretty well so far. I hope I can see this solution in action tomorrow. Should be possible, if i find a proper library to do the SVD algorithm... – xeed Mar 18 '13 at 01:26
  • About the QR factorization. I learned about that before, but can't remember that much. The only thing I remember was sth like: $R^{-1}b = b_{1}:b_{2}$ and $Qb_{1} = x$ isnt that always zero, if b is zero? I can't get the non zero solution with my knowledge. ^^ – xeed Mar 18 '13 at 01:36
  • try this previous answer – adam W Mar 18 '13 at 01:41
  • Well, i tried this solution. Works pretty well for computed UV Points, but doesn't work for me with real XYZ - UV Pairs. The resulting camera transformation is correct for points near the points in the calculation, but does not work at all for any point. I get a determinant of 0.0001 or sth.(and with multiplying i get wrong positions (tx), which is very wrong. I haven't figured out how to improve the results yet. – xeed Mar 26 '13 at 13:03
  • I think you may still be trying to solve the singular system. The rank should be 11 and thus the determinant is zero. Thus you are not inverting a matrix. You are looking for the null space. If the matrix is invertible then there actually is no solution since then the null space is non-existent. – adam W Mar 26 '13 at 13:40
  • With real world data that has noise, the matrix will only be close to singular. You need the singular value decomposition. It will tell you your accuracy. If you have only one small singular value, then you are good. If you have more, then you need more data for better precision. – adam W Mar 26 '13 at 13:45
  • I actually used SVD to solve the problem. I just took the last column of the matrix as a solution. And as mentioned before, the resulting transformation matrix works for the points near the features. Thanks for the hint with the SVD. I'll work on it and see what informations I can gather. Furthormore you are right. I have a rank 12 matrix. – xeed Mar 27 '13 at 14:03
  • In addition to this solution I actually found a library to do the work for me. In opencv is a function called solvePNP(Ransac), which does everything I need, BUT until now I wasn't able to tweak the parameters so that I get a working solution. This speaks for the problem lying elsewhere. There are definetly some very wrong values in my pool, a few correct values and the major part is nearly correct. The ransac part of the function should kick out the wrong values, but what it currently does is kicking out many values and keeping 3 of (50-200) and 1 of them is wrong. ^^ – xeed Mar 27 '13 at 14:11
  • If you are interested: http://docs.opencv.org/trunk/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html#solvepnpransac – xeed Mar 27 '13 at 14:13