With
$$
\cases{
p = (x,y)\\
p_1=(x_1,y_1)\\
p_2=(x_2,y_2)\\
v_1 = (v_{x_1},v_{y_1})\\
u_1 = (-v_{y_1},v_{x_1})\\
v_2 = (v_{x_2},v_{y_2})\\
u_2 = (-v_{y_2},v_{x_2})\\
A = \left(\matrix{a& \frac b2\\ \frac b2& c}\right)\\
B = \left(\matrix{d & e}\right)
}
$$
and the ellipse $E(p) = p'Ap+Bp+f=0$ we have the conditions:
$$
\cases{
E(p_1)=0\\
E(p_2)=0\\
\nabla E(p_1) = \lambda_1 u_1\\
\nabla E(p_2) = \lambda_2 u_2\\
\det(A) \gt 0
}
$$
so we have $7$ equations and $8$ unknowns $(a,b,c,d,e,f,\lambda_1,\lambda_2)$.
From $\nabla E(p_i) = 2p_i'A + B = \lambda_i u_i$ we can obtain $\lambda_i$ easily. After dot right multiplication by $p_i$ we obtain
$$
\lambda_i = \frac{2p_i'Ap_i+Bp_i}{u_ip_i}
$$
thus we have
$$
\cases{
p_i'Ap_i+Bp_i + f = 0\\
\det(A) \gt 0\\
2p_i'A + B = \left(\frac{2p_i'Ap_i+Bp_i}{u_ip_i}\right)u_i\\
}\ \ \ \ \ \ \ (*)
$$
Focusing a case study having
$$
\cases{
p_1=(1,1)\\
p_2=(3,2)\\
v_1 = (\frac 12,1)\\
u_1 = (-1,\frac 12)\\
v_2 = (-1,-\frac 15)\\
u_2 = (\frac 15,-1)\\
\det(A) =a c-\frac{b^2}{4}=1
}
$$
from $(*)$ we have the equations
$$
\left\{
\begin{array}{rcl}
a+b+c+d+e+f & = & 0\\
9 a+6 b+4 c+3 d+2 e+f & = & 0 \\
a c-\frac{b^2}{4}-1 & = & 0\\
-2 a-3 b-4 c-d-2 e & = & 0\\
2 a+3 b+4 c+d+2 e & = & 0\\
\frac{2}{7} (30 a+13 b+4 c+5 d+e) & = & 0\\
-\frac{3}{7} (30 a+13 b+4 c+5 d+e) & = & 0\\
\end{array}
\right.
$$
Note that two equations can be eliminated so now we have $(5)$ equations and $(6)$ unknowns $(a,b,c,d,e,f)$ so we will solve for $(a,b,c,d,e)$ obtaining $(a(f),b(f),c(f),d(f),e(f)$ as follows
$$
\left[
\begin{array}{ccccc}
a&b&c&d&e\\
\frac{1}{37} \left(19 f-6 \sqrt{9 f^2-37}\right) & \frac{2}{37} \left(13 \sqrt{9 f^2-37}-35 f\right) & \frac{2}{37} \left(41 f-11 \sqrt{9
f^2-37}\right) & \frac{2}{37} \left(10 f-9 \sqrt{9 f^2-37}\right) & \frac{4}{37} \left(5 \sqrt{9 f^2-37}-22 f\right) \\
\end{array}
\right]
$$
Here we can choose $f$ to define the best fit. Follows a plot shoving the solutions parameterized by $f$.
