Here is one approach which is far away from the direction you started, but which gets you the answer pretty quickly. At least once you know the theory; explaining that is what will make this post a bit longer.
One can construct (or compute) the foci of a conic as the points where the tangents to the ideal circle points intersect. The ideal circle points are points with homogeneous coordinates $I=[1:i:0]$ and $J=[1:-i:0]$, so they lie on the line at infinity and have complex coordinates. These points lie on every circle, and therefore form the basis of Euclidean geometry in a projective setup. Conversely you can say that a given point is a focus if the lines connecting the point to the points at infinity is a tangent to the conic. So by knowing the two foci, you know four tangents. Plus you know the $y$ axis is a tangent as well, therefore you have five tangents which define the conic.
There is a concept of a dual conic, which is a conic defined not as a set of incident points but tangent lines. Just like a conic is uniquely defined by five incident points, no four of them collinear, its dual is uniquely defined by five tangent lines, no four of them concurrent. The math is exactly the same. To go from the matrix of the dual to that of the primal, you simply compute the adjunct (or inverse, but that's more work).
I'll denote your foci as $A=[-2:4:1]$ and $B=[4:6:1]$.
You might start by explicitely writing down your five defining lines computed from these points, using the cross product to compute the join of two points.
$$A\times I,\quad A\times J,\quad B\times I,\quad B\times J,\quad [1:0:0]$$
But the next step would be forming degenerate conics from these. In the primal situation that would mean picking four of the five points, forming two pairs of two points each, computing the line through the points from each pair and then computing the symmetric product matrix from these. In the dual world we have here, you'd be intersecting lines instead as your first step. But we already know the points of intersection, at least four of them! So you can already build two degenerate conics from these.
\begin{align*}
M_1&=\frac12\left(A\cdot B^T+B\cdot A^T\right)\\&=
\frac12\begin{pmatrix}-2\\4\\1\end{pmatrix}\cdot(4,6,1)+
\frac12\begin{pmatrix}4\\6\\1\end{pmatrix}\cdot(-2,4,1)\\&=
\frac12\begin{pmatrix}
-8&-12&-2\\
16&24&4\\
4&6&1
\end{pmatrix}+\frac12\begin{pmatrix}
-8&16&4\\
-12&24&6\\
-2&4&1
\end{pmatrix}\\&=\begin{pmatrix}
-8&2&1\\
2&24&5\\
1&5&1
\end{pmatrix}
\\
M_2&=\frac12\left(I\cdot J^T+J\cdot I^T\right)=\begin{pmatrix}
1&0&0\\
0&1&0\\
0&0&0
\end{pmatrix}
\end{align*}
These matrices represent degenerate dual conics. That means a “conic” which has as tangents all the lines which pass through either of the constituent points. This means that the first four of the five defining points will be tangents to both of them, and therefore also to linear combinations of them. Now you only need to choose the linear combination in such a way as to make $g=[1:0:0]$, the $y$ axis, a tangent as well.
\begin{align*}
M &= (g^T\cdot M_2\cdot g)M_1 - (g^T\cdot M_1\cdot g)M_2
\\&= M_1 + 8M_2 = \begin{pmatrix}
0&2&1\\
2&32&5\\
1&5&1
\end{pmatrix}
\end{align*}
Now you have the dual conic. In order to obtain the primal one. To get that, you compute the adjunct of this matrix, by computing all $2\times 2$ subdeterminants (minors).
$$N=\operatorname{adj}(M)=\begin{pmatrix}
\begin{vmatrix}32&5\\5&1\end{vmatrix} &
-\begin{vmatrix}2&5\\1&1\end{vmatrix} &
\begin{vmatrix}2&32\\1&5\end{vmatrix} \\
-\begin{vmatrix}2&1\\5&1\end{vmatrix} &
\begin{vmatrix}0&1\\1&1\end{vmatrix} &
-\begin{vmatrix}0&2\\1&5\end{vmatrix} \\
\begin{vmatrix}2&1\\32&5\end{vmatrix} &
-\begin{vmatrix}0&1\\2&5\end{vmatrix} &
\begin{vmatrix}0&2\\2&32\end{vmatrix}
\end{pmatrix}=\begin{pmatrix}
7 & 3 & -22 \\
3 & -1 & 2 \\
-22 & 2 & -4
\end{pmatrix}$$
So the equation of your conic is
$$(x,y,1)\cdot\begin{pmatrix}
7 & 3 & -22 \\
3 & -1 & 2 \\
-22 & 2 & -4
\end{pmatrix}\cdot\begin{pmatrix}x\\y\\1\end{pmatrix}
=7x^2-y^2+6xy-44x+4y-4=0$$
I guess all of this sounded pretty complicated upon first reading. But take another look at what you actually had to compute. The matrix construction for $M_1$ and $M_2$ was fairly easy: $M_1$ is essentially just a clever combination of the coordinates of the foci, while $M_2$ will always be this matrix since $I$ and $J$ have fixed coordinates. The quadratic forms vector times matrix times vector are rather simple, and even more so if the vector of the fifth line is as simple as in this case. Computing the adjunct is elementary arithmetic in fancy notation. At the bottom of it, all the operations you needed were addition, subtraction and multiplication. You wouldn't even have needed the division from the $\frac12$ in the symmetric product; you could use twice these matrices just as well. You don't need square roots, you don't need case distinctions. You don't need to know that your conic is a hyperbola and not an ellipse. On the whole I think this makes the presented solution worth learning a bit of the theory.
Let's also compare my solution with what you had so far.
\begin{align*}
\frac{(3x+y-8)^2}{(3^2+1^2)a^2}-\frac{(x-3y+14)^2}{(1^2+3^2)b^2}&=1\\
b^2(3x+y-8)^2-a^2(x-3y+14)^2&=10a^2b^2\\
(-a^2 + 9b^2)x^2+(-9a^2+b^2)y^2+(6a^2+6b^2)xy\qquad\\
+(-28a^2-48b^2)x+(84a^2-16b^2)y+(-10a^2b^2-196a^2+64b^2)&=0\\
a^2=2\qquad b^2=8\qquad\\
70x^2-10y^2+60xy-440x+40y-40&=0
\end{align*}
This is exactly $10$ times the equation I derived above. So if you want to, you can give your hyperbola as
$$\frac{(3x+y-8)^2}{20}-\frac{(x-3y+14)^2}{80}=1$$
