I would like to give a simple solution based on basic algebraic principles of quadratic equations.
Let, $\frac{A^2 +B^2}{AB+1} = K$ and B = XA, where, X is a rational number (as both A and B are integers).
Then,
$\frac{A^2 +B^2}{AB+1}$ = $\frac{A^2 + A^2X^2}{XA^2 +1}$ = $K$
This gives the following quadratic equation in X,
$A^2X^2$ -$A^2KX$ + $A^2 - K$ = $0$
As X is a rational number, sum ($A^4K^2$ - $4A^2(A^2-K)$ ) is either a perfect square or equal to zero.
(If the solution of a quadratic equation, $ax^2$ + $bx$ + $c$ is a rational number, then the term $b^2$ - $4ac$ must be either a perfect square or equal to zero)
Case 1: ($A^4K^2$ - $4A^2(A^2-K)$ ) is equal to zero
$\Rightarrow$ $A^4K^2$ + $4A^2K$ -$4A^4$ = $ 0$
$\Rightarrow$ $K$ = $ \frac{-2 \pm 2 \sqrt{2}}{A^2} $
This is not a desired solution, as K is an integer.
Case 2: ($A^4K^2$ - $4A^2(A^2-K)$ ) is a perfect square
$A^4K^2$ - $4A^2(A^2-K)$ is a quadratic equation in $A^2$
Therefore, $(K^2 - 4)A^2$ +$4KA^2$ is a perfect square,
$\Rightarrow$ Either $16K^2$ = $0$ or $4K$ = $0$(If the quadratic equation, $ax^2$ + $bx$ + $c$ is a perfect square, then one possibility is ($b^2$ - $4ac$) is equal to zero, the other possibility is both $b$ and $c$ are equal to zero)
$\Rightarrow$ $K$ = $0$ ; This is not a desired solution.
Reconsidering the equation $(K^2 - 4)A^2$ +$4KA^2$ and writing it in the following way,
$A^4K^2$ - $4A^2(A^2-K)$
Now, as we have exhausted the other alternatives, this equation is a perfect square only if the second term is equal to zero, or $4A^2(A^2-K)$ = $0$.
$\Rightarrow$ $A^2$ = $K$ $\Rightarrow$ $B$ = $A^3$ (This completes the proof of the preliminary question!)
Let M be an even integer such that M = 2A,
$\Rightarrow$ $K$ = $\frac{M^2}{4}$ and $B$ = $\frac{M^3}{8}$ (Hence both the solution sets as proposed by jllbones are the same!)
Given integers $$ M \geq m > 0, $$ along with positive integers $x,y$ with $$ x^2 - Mxy + y^2 = m. $$ Then $m$ is a square.
– Will Jagy Aug 26 '19 at 23:54