We have
$$I = \int_0^{\infty} \dfrac{\log^2(x)}{1+x^2} dx = \int_0^1 \dfrac{\log^2(x)}{1+x^2} dx + \int_1^{\infty} \dfrac{\log^2(x)}{1+x^2} dx$$
Now $$\overbrace{\int_1^{\infty} \dfrac{\log^2(x)}{1+x^2} dx = -\int_1^0 \dfrac{\log^2(x)}{1+x^2}dx}^{x \mapsto 1/x} = \int_0^1 \dfrac{\log^2(x)}{1+x^2}dx$$
Hence, we get that
$$I = 2 \int_0^1 \dfrac{\log^2(x)}{1+x^2}dx = 2 \sum_{k=0}^{\infty}(-1)^k \int_0^1 x^{2k} \log^2(x) dx$$
$$\int_0^1 x^{2k} \log^2(x) dx = \dfrac2{(2k+1)^3} \,\,\,\, (\spadesuit)$$
Hence,
$$I = 4 \sum_{k=0}^{\infty}\dfrac{(-1)^k}{(2k+1)^3} = \dfrac{\pi^3}8 \,\,\,\, (\clubsuit)$$
$(\spadesuit)$ is nothing but the $\Gamma$-function with some scaling factors.
$$I_k = \underbrace{\int_0^1 x^{2k} \log^2(x) dx = \int_0^{\infty} t^2 e^{-(2k+1)t}dt}_{x \mapsto e^{-t}}$$
Now let $(2k+1)t = x$. We then get
$$I_k = \int_0^{\infty} \dfrac{x^2}{(2k+1)^2} e^{-x} \dfrac{dx}{2k+1} = \dfrac{\Gamma(3)}{(2k+1)^3} = \dfrac2{(2k+1)^3}$$
$(\clubsuit)$ is evaluated as follows. We have
$$\text{Li}_s(z) = \sum_{k=1}^{\infty} \dfrac{z^k}{k^s}$$
Now $$\text{Li}_3(i) = \sum_{k=1}^{\infty} \dfrac{i^k}{k^s}$$
$$\text{Li}_3(-i) = \sum_{k=1}^{\infty} \dfrac{(-1)^ki^k}{k^s}$$
Hence,
$$\text{Li}_3(i) - \text{Li}_3(-i) = 2i \left(\sum_{k=0}^{\infty} \dfrac{(-1)^k}{(2k+1)^3}\right) \,\,\,\, (\heartsuit)$$
Now the PolyLogarithmic function satisfies a very nice identity
$$\text{Li}_n(e^{2 \pi ix}) + (-1)^n \text{Li}_n(e^{-2 \pi ix}) = - \dfrac{(2\pi i)^n}{n!}B_n(x)$$
Taking $n=3$ and $x=\dfrac14$, gives us
$$\text{Li}_3(i) - \text{Li}_n(-i) = - \dfrac{(2\pi i)^3}{3!}B_3(1/4) = i \dfrac{8 \pi^3}{6} \dfrac3{64} =i \dfrac{\pi^3}{16} \,\,\,\, (\diamondsuit)$$
Comparing $(\heartsuit)$ and $(\diamondsuit)$ gives us $(\clubsuit)$.