In this case, if $k=5,$ then we would have $70$ as the sum of two integer squares, which it is not.
$$ a^2 + b^2 = 5a + 5b + 5, $$
$$ 4a^2 + 4 b^2 = 20a + 20 b + 20, $$
$$ 4 a^2 - 20a + 4 b^2 - 20 b = 20, $$
$$ 4 a^2 - 20a + 25 + 4 b^2 - 20 b + 25 = 20 + 25 + 25 = 70, $$
$$ (2a-5)^2 + (2b-5)^2 = 70. $$
However, $70 = 2 \cdot 5 \cdot 7$ is not the sum of two integer squares.
A superficially similar problem that does lead to $5$ is
$$ \frac{a^2 + b^2}{ab-1} = k, $$
see Is it true that $f(x,y)=\frac{x^2+y^2}{xy-t}$ has only finitely many distinct positive integer values with $x$, $y$ positive integers? and my three or so answers there. I did, with help, eventually settle the whole thing.
If the version I have guessed is really the original problem, then it fits under https://en.wikipedia.org/wiki/Vieta_jumping