Here is one way to evaluate your integral (there are no doubt other ways).
Let
$$I = \int^\infty_0 \frac{\ln^2 x}{a^2 + x^2} \, dx, \quad a > 0.$$
Setting $x = \dfrac{a}{u}$ then $dx = -\dfrac{a}{u^2} \, du$ and we have
\begin{align*}
I &= a \int^\infty_0 \frac{\ln^2 \left (\frac{a}{u} \right )}{\frac{a^2}{u^2} + a^2} \cdot \frac{du}{u^2}\\
&= \frac{1}{a} \int^\infty_0 \frac{\ln^2 \left (\frac{a}{u} \right )}{1 + u^2} \, du\\
&= \frac{1}{a} \int^\infty_0 \frac{\left [\ln a - \ln u \right ]^2}{1 + u^2} \, du\\
&= \frac{\ln^2 a}{a} \int^\infty_0 \frac{du}{1 + u^2} - \frac{2 \ln a}{a} \int^\infty_0 \frac{\ln u}{1 + u^2} \, du + \frac{1}{a} \int^\infty_0 \frac{\ln^2 u}{1 + u^2} \, du\\
&= \frac{\ln^2 a}{a} I_1 - \frac{2 \ln a}{a} I_2 + \frac{1}{a} I_3.
\end{align*}
Now evaluating each of these integrals. The first is trival. We have
$$I_1 = \int^\infty_0 \frac{du}{1 + u^2} = \left [\tan^{-1} u \right]^\infty_0 = \frac{\pi}{2}.$$
For the second, setting $u = \dfrac{1}{y}, du = -\dfrac{1}{y^2} \, dy$. Thus
$$I_2 = \int^\infty_0 \frac{\ln u}{1 + u^2} \, du = \int^\infty_0 \frac{\ln \left (\frac{1}{y} \right )}{1 + \frac{1}{y^2}} \cdot \frac{dy}{y^2} = - \int^\infty_0 \frac{\ln y}{1 + y^2},$$
or $I_2 = 0$.
For the third integral we begin by dividing the interval of integration as follows
$$I_3 = \int^\infty_0 \frac{\ln^2 u}{1 + u^2} \, du = \int^1_0 \frac{\ln^2 u}{1 + u^2} \, du + \int^\infty_1 \frac{\ln^2 u}{1 + u^2} \, du.$$
If in the rightmost integral we set $u = \dfrac{1}{y}, du = -\dfrac{1}{y^2} \, dy$, then
\begin{align*}
I_3 &= \int^1_0 \frac{\ln^2 u}{1 + u^2} \, du + \int^1_0 \frac{\ln^2 \left (\frac{1}{y} \right )}{1 + \frac{1}{y^2}} \cdot \frac{dy}{y^2}\\
&= \int^1_0 \frac{\ln^2 u}{1 + u^2} \, du + \int^1_0 \frac{\ln^2 y}{1 + y^2} \, dy\\
&= 2 \int^1_0 \frac{\ln^2 u}{1 + u^2} \, du.
\end{align*}
Noting the term $1/(1 + u^2)$ can be written as the sum of a geometric series, namely
$$\frac{1}{1 + u^2} = \sum^\infty_{n = 0} (-1)^n u^{2n}, \quad |u| < 1$$
replacing the term in the integral with this sum, after interchanging the integral sign with the summation we have
$$I_3 = 2 \sum^\infty_{n = 0} (-1)^n \int^1_0 u^{2n} \ln^2 u \, du.$$
After performing integration by parts twice, we are left with
$$I_3 = 4 \sum^\infty_{n = 0} \frac{(-1)^n}{(2n + 1)^3}.$$
From the definition for the Dirichlet beta function $\beta (s)$, namely
$$\beta (s) = \sum^\infty_{n = 0} \frac{(-1)^n}{(2n + 1)^s},$$
we recognise our sum as corresponding to $\beta (3)$. Thus
$$I_3 = 4 \beta (3).$$
The value for $\beta (3)$ can be readily found (see here for example). It is
$$\beta (3) = \frac{\pi^3}{32},$$
giving
$$I_3 = 4 \cdot \frac{\pi^3}{32} = \frac{\pi^3}{8}.$$
So we finally have
$$I = \frac{\ln^2 a}{a} \cdot \frac{\pi}{2} + \frac{1}{a} \cdot \frac{\pi^3}{8},$$
or
$$\int^\infty_0 \frac{\ln^2 x}{a^2 + x^2} \, dx = \frac{\pi}{2a} \left (\ln^2 a + \frac{\pi^2}{4} \right ).$$