After the discussion in the comments, I assume that the proportions of the desired rectangle are known to be $a:b$.
Though the method I describe below is good for understanding, it is not suitable for direct implementation. The resulting equation system is typically ill-conditioned. (It is also possible to get rid of the $\lambda_i$ by using cross products to express collinearity, and thus get a slightly smaller system to solve. This is described in detail in Multiple View Geometry by Hartley and Zisserman.)
Let $\boldsymbol{p}_i = (x_i, y_i, 1)$ be the homogeneous coordinates of the four corners in the input image. Let their desired destinations be
$$
\left\{
\begin{aligned}
\boldsymbol{q}_1 = (0, 0, 1) \\
\boldsymbol{q}_2 = (a, 0, 1) \\
\boldsymbol{q}_3 = (a, b, 1) \\
\boldsymbol{q}_4 = (0, b, 1)
\end{aligned}
\right..
$$
The corners are related through a planar homography, represented by a $3\times 3$-matrix $\boldsymbol{H}$, and the relations are $\lambda_i\boldsymbol{q}_i=\boldsymbol{H}\boldsymbol{p}_i$ for some scalars $\lambda_i$.
How does one determine $\boldsymbol{H}$? Let $\boldsymbol{h}_1^T$, $\boldsymbol{h}_2^T$ and $\boldsymbol{h}_3^T$ be the rows of $\boldsymbol{H}$. Then
$$
\left\{
\begin{aligned}
\lambda_i q_{i1} = \boldsymbol{h}_1^T\boldsymbol{p}_i \\
\lambda_i q_{i2} = \boldsymbol{h}_2^T\boldsymbol{p}_i \\
\lambda_i q_{i3} = \boldsymbol{h}_3^T\boldsymbol{p}_i
\end{aligned}
\right.
\Longleftrightarrow
\left\{
\begin{aligned}
\boldsymbol{p}_i^T\boldsymbol{h}_1 - \lambda_i q_{i1} = 0 \\
\boldsymbol{p}_i^T\boldsymbol{h}_2 - \lambda_i q_{i2} = 0 \\
\boldsymbol{p}_i^T\boldsymbol{h}_3 - \lambda_i q_{i3} = 0
\end{aligned}
\right..
$$
Write this system of equations in matrix form,
$$\begin{bmatrix}
\boldsymbol{p}_1^T & \boldsymbol{0} & \boldsymbol{0} & q_{11} & 0 & 0 & 0 \\
\boldsymbol{0} & \boldsymbol{p}_1^T & \boldsymbol{0} & q_{12} & 0 & 0 & 0 \\
\boldsymbol{0} & \boldsymbol{0} & \boldsymbol{p}_1^T & q_{13} & 0 & 0 & 0 \\
\boldsymbol{p}_2^T & \boldsymbol{0} & \boldsymbol{0} & 0 & q_{21} & 0 & 0 \\
\boldsymbol{0} & \boldsymbol{p}_2^T & \boldsymbol{0} & 0 & q_{22} & 0 & 0 \\
\boldsymbol{0} & \boldsymbol{0} & \boldsymbol{p}_2^T & 0 & q_{23} & 0 & 0 \\
\boldsymbol{p}_3^T & \boldsymbol{0} & \boldsymbol{0} & 0 & 0 & q_{31} & 0 \\
\boldsymbol{0} & \boldsymbol{p}_3^T & \boldsymbol{0} & 0 & 0 & q_{32} & 0 \\
\boldsymbol{0} & \boldsymbol{0} & \boldsymbol{p}_3^T & 0 & 0 & q_{33} & 0 \\
\boldsymbol{p}_4^T & \boldsymbol{0} & \boldsymbol{0} & 0 & 0 & 0 & q_{41} \\
\boldsymbol{0} & \boldsymbol{p}_4^T & \boldsymbol{0} & 0 & 0 & 0 & q_{42} \\
\boldsymbol{0} & \boldsymbol{0} & \boldsymbol{p}_4^T & 0 & 0 & 0 & q_{43} \\
\end{bmatrix}
\begin{bmatrix}
\boldsymbol{h}_1 \\
\boldsymbol{h}_2 \\
\boldsymbol{h}_3 \\
\lambda_1 \\
\lambda_2 \\
\lambda_3 \\
\lambda_4
\end{bmatrix}
= \boldsymbol{0},
$$
and solve. Now, any point $\boldsymbol{p}$ in the input image is mapped to the point $\lambda\boldsymbol{q}=\boldsymbol{H}\boldsymbol{p}$. In particular, the corners are mapped to the corners of a rectangle.