$a$ and $b$ are positive integers such that $$\frac{a^2+b^2+1}{ab+1}=k$$ where $k$ is positive integer. Find all values $k$
$a$ is the root of the equation $$x^2-kbx+b^2+1-k=0$$ Let $c$ is another root of this equation. Then $c=kb-a$ and $ac=b^2+1-k$
Note that, mod $ab+1$, $0=b^2(a^2+b^2+1)=b^4+b^2+1$. Therefore, $b^4+b^2+1 > 0$ is divisible by $ab+1$, thus $b(b^3+b) \geq ab$. So if $b \neq 0$, then this can be written as $k \leq b^2+1$ which entails (if $a > 0$), $c \geq 0$.
Take now, for a given $k$, $(a,b)$ a solution in non-negative integers with $a \geq b$ and $a+b$ minimal. Assume $b \neq 0$ (hence $ab>0$).
Then by minimality $c \geq a$ (since $(c,b)$ is a solution) thus $a^2 \leq ac = b^2+1-k \leq b^2$, so $a=b$. But this implies $a=b=0$, a contradiction. So $b=0$ and $k=a^2+1$.
The possible values of $k$ are thus exactly all the $x^2+1$ for integers $x$.
Let $k$ be a positive integer such that there exist positive integers $a$ and $b$ for which $$\frac{a^2+b^2+1}{ab+1}=k\tag{1}$$ Suppose that $(a,b)=(p,q)$ is the smallest positive integer answer to $(1)$ such that $p\geq q$ (in the sense that $p+q$ is the least possible). Therefore, $t=p$ is a solution to $$t^2-(kq)t+(q^2+1-k)=0.\tag{2}$$ This is a quadratic equation so there is another root $t=r$. Note that $$r=kq-p$$ so $r$ is an integer.
Suppose that $r<0$. Let $f_0(x)=1$ and $g_0(x)=1$. We want to show that for any non-negative integer $j$, there are polynomials $f_j(x),g_j(x)\in\mathbb{Z}[x]$ with non-negative integer coefficients such that $$k\geq f_j(q)\geq 3j$$ and $$p\geq g_j(q)\geq 3j+1.$$ Furthermore, $f_j$ has degree $j+1$ and $g_j$ has degree $j+2$ for $j>0$.
The case $j=0$ is trivial. We proceed further by induction. Suppose that $f_j(x),g_j(x)$ are known and satisfy the requirements. Then, we have from $(2)$ that $pr=q^2+1-k$ or $$k-q^2-1=p(-r)\geq p\geq g_j(q).$$ Therefore, $k\geq q^2+1+g_j(q)$ and we can define $$f_{j+1}(x)=x^2+1+g_j(x).$$ From $r=kq-p$, we get $kq-p<0$ or $p>kq$. That is, $p\geq kq+1$. Because $k\geq f_{j+1}(q)$, we have $$p\geq q\ f_{j+1}(q)+1.$$ Hence, we define $$g_{j+1}(x)=x\ f_{j+1}(x)+1=x^3+x+1+x\ g_j(x).$$ Note that this construction guarantees that $f_{j+1}$ and $g_{j+1}$ has non-negative integer coefficients, $$\deg f_{j+1}=\max\{2,\deg g_j\},$$ and $$\deg g_{j+1}=\deg f_j+1=\max\{2,\deg g_j\}+1.$$ It follows that $$\deg f_j=j+1$$ and $$\deg g_j=j+2$$ for every positive integer $j$. It can be easily seen that $$f_j(q)\geq f_j(1)=\max\{1,3j\}$$ and $$g_j(q)\geq g_j(1)=3j+1$$ for all positive integers $q$ and for every non-negative integer $j$.
From the result above, if $r<0$, then $k$ and $p$ are arbitrarily large, which is a contradiction. Therefore, we must have $r\geq 0$. We want to show that $r>0$ is also impossible.
Suppose for the sake of contradiction that $r>0$. Then, $(a,b)=(q,r)$ is also a solution to $(1)$. By minimality of $(p,q)$, $p+q\leq q+r$, or $p\leq r$. That means $$kq=p+r\leq 2r.$$ Hence, $r\geq \frac{kq}{2}$, so $$q^2+1-k=pr\geq p\left(\frac{kq}{2}\right)=\frac{kpq}{2}.$$ Therefore, $$q^2+1\geq \frac{k}{2}(pq+2)\geq \frac{k}{2}(q^2+2),$$ since $p\geq q$. That is, $$k\leq 2\left(\frac{q^2+1}{q^2+2}\right)< 2.$$ Therefore, $k=1$. However, by $(1)$, AM-GM implies $$1=\frac{p^2+q^2+1}{pq+1}\geq \frac{2pq+1}{pq+1}>\frac{pq+1}{pq+1}=1,$$ which is absurd. Therefore, $r>0$ cannot happen.
This means $r=0$, making $k=q^2+1$. In this case, there always exists a solution with the minimal solution $(a,b)$ such that $a\geq b$ being $$(a,b)=(q^3+q,q).$$ All integer solutions $(a,b)$ such that $a\geq b >0$ satisfies $$(a,b)=\left(x_{s+1}^q,x_s^{q}\right)$$ for some positive integer $s$, where the sequence $\left(x_s^q\right)$ is defined by $$x_1^q=q,$$ $$x_2^q=q^3+q,$$ $$x_s^q=(q^2+1)x_{s-1}^q-x_{s-2}^q$$ for $s=3,4,5,\ldots$.
