I have a rectangle which is transformed into another one by an affine transformation:
$$ \begin{bmatrix}x \\ y\end{bmatrix} \mapsto \begin{bmatrix}m_1 & m_2 \\ m_3 & m_4\end{bmatrix} \cdot\begin{bmatrix}x \\ y\end{bmatrix} +\begin{bmatrix}t_x \\ t_y\end{bmatrix} $$
I want to determine this transformation. However, I have 4 points and 6 unknowns. Is there any condition I can fix to reduce the number of unknowns? (like, for instance, no reflection?)
How would that translate?
You have 4 points in the plane, each with two coordinates, which amounts to 8 conditions. So there is no need to reduce the number of unknowns; in fact you have an overconstrained system. An affine transformation is already uniquely defined by three points and their images, as long as these are not on a common line.
You might use a least squares approach to find those parameters which best match your data, but that will yield an exact match only if your rectangle is guaranteed to be transformed only by an affine transform.
Mapping four points to four other in a general setup can be achieved using a projective transformation.
