The equation of the desired ellipse is of the form
$ (r - r_0)^T Q (r - r_0) = 1 \hspace{26pt}(1) $
which is also equivalent to the following form
$ r^T Q r + r^T b + c = 0 \hspace{26pt}(2)$
where $ r = [x, y]^T $, $r_0$ is the center of the ellipse, $Q$ is a $2 \times 2 $ positive definite symmetric matrix. In the second form we have $ b = - 2 Q r_0 $ and $ c = r_0^T Q r_0 - 1 $.
Equation $(2)$ can be written compactly as follows:
$ w^T Q_0 w = 0 \hspace{26pt}(3)$
where $ w = \begin{bmatrix} r \\ 1 \end{bmatrix} $ is a $3 \times 1$ vector. And the $3 \times 3 $ matrix $Q_0$ is specified by
$ Q_0 = \begin{bmatrix} Q && \dfrac{1}{2}b \\ \dfrac{1}{2} b^T && c \end{bmatrix} \tag{4} $
Using the methods of projective geometry, it is known that if the line
$ u_1 x + u_2 y + u_3 = 0 $
is tangent to the conic specified by $(3)$ then $ u = [u_1, u_2, u_3]^T$ satifies,
$$ u^T Q_0^{-1} u = 0 \tag{5}$$
And this is our starting point. Using the five given vertices of the pentagon, we'll construct the equations of five sides of the pentagon, and use those in equation (5), to determine $Q^{-1}$ up to a scalar multiple.
Detemining the equation of the line between vertex $P_i$ and $P_{i+1}$ is performed as follows. Define the vector
$ v_i = P_{i+1} - P_i $
Then the equation of the line connecting $P_{i} $ with $P_{i+1}$ is
$ (E v_i)^T ( \mathbf{r} - P_i ) = 0 $
where $ E = \begin{bmatrix} 0 && 1 \\ -1 && 0 \end{bmatrix} $
So that the first two entries of vector $u_i$ are given by $E v_i$ and the third entry is $( - (E v_i)^T P_i ) $.
So for each of the five sides, we have thus calculated the vector $u_i, i = 1, 2, 3, 4, 5$
The quadratic form given by equation $(5)$ when expanded becomes:
$ A_{11} u_1^2 + A_{22} u_2^2 + A_{33} u_3^2 + 2 A_{12} u_1 u_2 + 2 A_{13} u_1 u_3 + 2 A_{23} u_2 u_3 = 0 \tag{6}$
where matrix $A = Q_0^{-1}$
There are six unknowns in this equation, which are the $A_{ij}'s$. Using the five $u_i$'s for the five sides, we can write $5$ of these equations. The solution of this $5 \times 6$ system is as follows
$ X = [A_{11}, A_{22}, A_{33}, A_{12}, A_{13}, A_{23} ] = \lambda X_0 $
The scalar $\lambda$ does not affect the calculation of $Q$, and can be taken as unity.
We thus have matrix $A$, from which we calculate $Q_0 = A^{-1}$. This $Q_0$ is a scalar multiple of the actual $Q_0$. That is,
$ Q_0 = \begin{bmatrix} s Q && \dfrac{1}{2} s b \\ \dfrac{1}{2} s b^T && s c \end{bmatrix} $
The scalar $s$ is unknown. Using matrix $Q_0$, we can find the center of the ellipse as follows
$ r_0 = - \dfrac{1}{2} (s Q) ^{-1} (s b) = - \dfrac{1}{2} Q ^{-1} b$
Note that in the above calculation we're using $(sQ)$ and $(sb)$ which we have.
Having found the center $r_0$, the equation of the ellipse is
$ (r - r_0)^T (s Q) (r - r_0) = s c + r_0^T(s Q) r_0 $
Therefore,
$ (r - r_0)^T \left( \dfrac{ s Q }{ s c + r_0^T (s Q) r_0 } \right) (r - r_0) = 1 $
Comparing this to
$ (r - r_0)^T Q (r - r_0) = 1 $
We get our $Q$ as
$ Q = \dfrac{ s Q }{ s c + r_0^T (s Q) r_0 } $
Finally to obtain the parameters of this ellipse, diagonalize $Q$ as follows
$ Q = R \ D \ R^T $
The diagonal entries of $D$ are the reciprocal of the square of the semi-major and semi-minor axes (in some order), and $R$ is the rotation matrix whose columns are unit vectors that point in the direction of these axes.
Applying the above method to the given pentagon, gives
$ \text{Semi-major axis} = a = 5.209482 $
$ \text{Semi-minor axis} = b = 3.830313 $
$ \text{Angle that major axis makes with positive } x \text{ axis} = \theta = 1.997836 = 114.4676^\circ$
$\text{Center of ellipse} = (10.9, 5) $
Finally the inscribed ellipse is plotted with the given pentagon. This is shown below.
