The trick to solving this integral is to reduce it to two power series and then evaluate the power series using differential equations. The final answer is
$$\int_{0}^{\infty}\frac{\ln\left(1+ax+x^{2}\right)}{1+x^{2}}dx=\left(\frac{1}{2}\sin^{-1}\left(\frac{a}{2}\right)-\frac{\pi}{4}\right)\ln\left(\frac{1-\sqrt{1-\frac{a^{2}}{4}}}{1+\sqrt{1-\frac{a^{2}}{4}}}\right)+\frac{\pi}{2}\ln\left|a\right|-\Delta_{\pi}\mathrm{Cl}_{2}\left(\sin^{-1}\left(\frac{a}{2}\right)\right)$$
Here, $\Delta_h$ is the forward difference operator defined by
$$\Delta_h[f](x)=f(x+h)-f(x)$$
And $\mathrm{Cl}_2(\theta)$ is the SL-type Clausen function, defined by
$$\mathrm{Cl}_2(\varphi)=\int_0^\varphi \ln\left|2\sin\left(\frac{x}{2}\right)\right|dx$$
If you want to compute answers to your integral, you can use the fact that
$$\Delta_{\pi}\mathrm{Cl}_{2}\left(\sin^{-1}\left(\frac{a}{2}\right)\right)=-2\sum_{k=0}^{\infty}\frac{\sin\left(\left(2k+1\right)\sin^{-1}\left(\frac{a}{2}\right)\right)}{\left(2k+1\right)^{2}}$$
To begin solving the integral, we substitute $x=\tan(\theta)$ to get that
\begin{align*}
I&=\int_{0}^{\infty}\frac{\ln\left(1+ax+x^{2}\right)}{1+x^{2}}dx\\
&=\int_{0}^{\frac{\pi}{2}}\ln\left(\sec^{2}\left(x\right)+a\tan\left(x\right)\right)dx\\
&=\int_{0}^{\frac{\pi}{2}}\ln\left(1+a\sin\left(x\right)\cos\left(x\right)\right)dx-2\int_{0}^{\frac{\pi}{2}}\ln\left(\cos\left(x\right)\right)dx\\
\end{align*}
We can now use the fact that $\sin(2x)=2\sin(x)\cos(x)$ and symmetries in the argument of $\sin(x)$ to get that
$$\int_{0}^{\frac{\pi}{2}}\ln\left(1+a\sin\left(x\right)\cos\left(x\right)\right)dx=\int_{0}^{\frac{\pi}{2}}\ln\left(1+\frac{a}{2}\sin\left(x\right)\right)dx$$
We will also note that it is very easy to show that
$$2\int_{0}^{\frac{\pi}{2}}\ln\left(\cos\left(x\right)\right)dx=-\pi\ln\left(2\right)$$
We can now substitute these two expressions into where we left off, preparing to expand $\ln(x)$ as a power series, to get that
\begin{align*}
I&=\int_{0}^{\frac{\pi}{2}}\ln\left(1+\frac{a}{2}\sin\left(x\right)\right)dx+\pi\ln\left(2\right)\\
&=-\int_{0}^{\frac{\pi}{2}}\sum_{n=1}^{\infty}\frac{\left(-\frac{a}{2}\sin\left(x\right)\right)^{n}}{n}dx+\pi\ln\left(2\right)\\
&=-\sum_{n=1}^{\infty}\frac{\left(-1\right)^{n}}{n}\left(\frac{a}{2}\right)^{n}\int_{0}^{\frac{\pi}{2}}\left(\sin\left(x\right)\right)^{n}dx+\pi\ln\left(2\right)
\end{align*}
We can now use the well known result that
$$\int_{0}^{\frac{\pi}{2}}\left(\sin\left(x\right)\right)^{n}dx=\frac{\sqrt{\pi}\Gamma\left(\frac{n+1}{2}\right)}{2\Gamma\left(\frac{n}{2}+1\right)}$$
where $\Gamma(x)=(x-1)!$ is the gamma function to get that
\begin{align*}
I&=-\frac{\sqrt{\pi}}{2}\sum_{n=1}^{\infty}\frac{\left(-1\right)^{n}}{n}\left(\frac{a}{2}\right)^{n}\frac{\Gamma\left(\frac{n+1}{2}\right)}{\Gamma\left(\frac{n}{2}+1\right)}+\pi\ln\left(2\right)\\
&=-\frac{\sqrt{\pi}}{4}\sum_{n=1}^{\infty}\frac{1}{n}\left(\frac{a}{2}\right)^{2n}\frac{\Gamma\left(n+\frac{1}{2}\right)}{\Gamma\left(n+1\right)}+\frac{\sqrt{\pi}}{2}\sum_{n=1}^{\infty}\frac{1}{2n+1}\left(\frac{a}{2}\right)^{2n+1}\frac{\Gamma\left(n+1\right)}{\Gamma\left(n+1+\frac{1}{2}\right)}+\pi\ln\left(2\right)
\end{align*}
Where the last equality was obtained by summing over even/odd numbers. We can now use the Legendre duplication formula, namely
$$\Gamma\left(n+\frac{1}{2}\right)=\sqrt{\pi}2^{1-2n}\frac{\Gamma\left(2n\right)}{\Gamma\left(n\right)}$$
to get both of the series in terms of factorials, namely
$$I=-\frac{\pi}{4}\sum_{n=1}^{\infty}\frac{1}{n}\frac{\left(2n\right)!}{\left(n!\right)^{2}}\left(\frac{a}{4}\right)^{2n}+\frac{1}{2}\sum_{n=0}^{\infty}\frac{1}{\left(2n+1\right)^{2}}\frac{\left(n!\right)^{2}}{\left(2n\right)!}a^{2n+1}+\pi\ln\left(2\right)$$
The first power series we turn our attention to is
$$\sum_{n=1}^{\infty}\frac{\left(2n\right)!}{\left(n!\right)^{2}}x^{n}$$
We can use the fact that
$$\frac{\left(2(n+1)\right)!}{\left((n+1)!\right)^{2}}=\frac{\left(2n\right)!}{\left(n!\right)^{2}}\cdot\left(\frac{4n+2}{n+1}\right)$$
to set up an easily solvable differential equation, which yields
$$\sum_{n=1}^{\infty}\frac{\left(2n\right)!}{\left(n!\right)^{2}}x^{n}=\frac{1}{\sqrt{1-4x}}-1$$
And thus by integrating
$$\sum_{n=1}^{\infty}\frac{1}{n}\frac{\left(2n\right)!}{\left(n!\right)^{2}}x^{n}=\ln\left(\frac{1-\sqrt{1-4x}}{1+\sqrt{1-4x}}\right)-\ln\left(x\right)$$
Substituting back in yields
$$I=-\frac{\pi}{4}\ln\left(\frac{1-\sqrt{1-\frac{a^{2}}{4}}}{1+\sqrt{1-\frac{a^{2}}{4}}}\right)+\frac{1}{2}\sum_{n=0}^{\infty}\frac{1}{\left(2n+1\right)^{2}}\frac{\left(n!\right)^{2}}{\left(2n\right)!}a^{2n+1}+\frac{\pi}{2}\ln\left(a\right)$$
Similarly, it can be shown that
$$\sum_{n=0}^{\infty}\frac{\left(n!\right)^{2}}{\left(2n\right)!}x^{2n+1}=\frac{4\left(x\sqrt{4-x^{2}}+\sin^{-1}\left(\frac{x}{2}\right)x^{2}\right)}{\left(4-x^{2}\right)^{\frac{3}{2}}}$$
Which upon integrating once yields
$$\sum_{n=0}^{\infty}\frac{1}{2n+1}\frac{\left(n!\right)^{2}}{\left(2n\right)!}x^{2n+1}=\frac{4\sin^{-1}\left(\frac{x}{2}\right)}{\sqrt{4-x^{2}}}$$
And then integrating once more and substituting into our equation yields the final solution. If you have any questions about any of the steps, since I skipped over a lot, you can ask me.
\displaystylein titles. – StubbornAtom Mar 23 '20 at 16:53