By the residue theorem
$$\oint_{C_1 \cup C_2 \cup C_3 \cup C_4}f(z) \, dz = 2\pi i Res(f,i) = 2\pi i (i \pi^2/8) = -\frac{\pi^3}{4}.$$
It is straightforward to show that the integrals over $C_2$ and $C_4$ vanish in the limit as $R \to \infty$ and $\rho \to 0$.
Hence,
$$\tag{1} -\frac{\pi^3}{4} = \int_0^\infty \frac{\ln^2 x}{1 + x^2} \, dx + \lim_{R \to \infty, \rho \to 0} \int_{C_1} \frac{\ln^2 z}{1 + z^2} \, dz.$$
Now we can focus on the second integral on the RHS -- the source of your difficulty.
Note that
$$\begin{align}\lim_{R \to \infty, \rho \to 0} \int_{C_1} \frac{\ln^2 z}{1 + z^2} \, dz &= \int_{-\infty}^0 \frac{ \ln^2 x}{1 + x^2} \, dx \\ &= \int_{0}^\infty \frac{ \ln^2 (-x)}{1 + x^2} \, dx \\ &= \int_{0}^\infty \frac{ (\ln x + i \pi)^2}{1 + x^2} \, dx \\ &= \int_{0}^\infty \frac{ \ln^2 x }{1 + x^2} \, dx + 2\pi i\int_{0}^\infty \frac{ \ln x }{1 + x^2} \, dx - \pi^2\int_{0}^\infty \frac{ 1 }{1 + x^2} \, dx \\ &= \int_{0}^\infty \frac{ \ln^2 x }{1 + x^2} \, dx + 0 - \pi^2 \frac{\pi}{2} \\ &= \int_{0}^\infty \frac{ \ln^2 x }{1 + x^2} \, dx - \frac{\pi^3}{2}\end{align}.$$
The second of the three integrals on the RHS can be shown to be $0$ by showing that integrals over $[0,1]$ and $[1,\infty)$ must cancel (using a change of variables $u = 1/x)$.
Substituting into (1) we obtain
$$-\frac{\pi^3}{4} = 2\int_0^\infty \frac{\ln^2 x}{1 + x^2} \, dx - \frac{\pi^3}{2}.$$
Thus,
$$\int_0^\infty \frac{\ln^2 x}{1 + x^2} \, dx = \frac{\pi^3}{8}.$$
