Without Contour Integration
Note that
$$
\begin{align}
\int_0^1x^n\log(x)^2\,\mathrm{d}x
&=\frac1{n+1}\int_0^1\log(x)^2\,\mathrm{d}x^{n+1}\\
&=-\frac2{n+1}\int_0^1x^n\log(x)\,\mathrm{d}x\\
&=-\frac2{(n+1)^2}\int_0^1\log(x)\,\mathrm{d}x^{n+1}\\
&=\frac2{(n+1)^2}\int_0^1x^n\,\mathrm{d}x\\[3pt]
&=\frac2{(n+1)^3}
\end{align}
$$
Therefore, using the value of $\beta(3)$ from this answer,
$$
\begin{align}
\int_0^\infty\frac{\log(x)^2}{x^2+1}\,\mathrm{d}x
&=2\int_0^1\frac{\log(x)^2}{x^2+1}\,\mathrm{d}x\\
&=2\sum_{k=0}^\infty\int_0^1(-1)^kx^{2k}\log(x)^2\,\mathrm{d}x\\
&=\sum_{k=0}^\infty(-1)^k\frac4{(2k+1)^3}\\[6pt]
&=4\beta(3)\\[9pt]
&=\frac{\pi^3}8
\end{align}
$$
Using Contour Integration
Let
$$
\begin{align}
\gamma&=\left[(-1+i)\epsilon,Re^{i\sin^{-1}\left(\frac\epsilon{R}\right)}\right]\,\cup\,\color{#C00}{Re^{i\left[\sin^{-1}\left(\frac\epsilon{R}\right),2\pi-\sin^{-1}\left(\frac\epsilon{R}\right)\right]}}\\
&\cup\,\left[Re^{-i\sin^{-1}\left(\frac\epsilon{R}\right)},(-1-i)\epsilon\right]\,\cup\,\color{#090}{[(-1-i)\epsilon,(-1+i)\epsilon]}
\end{align}
$$
then, because the integral over the large red circular arc is bounded by $\color{#C00}{2\pi R\frac{(\log(R)+2\pi)^3}{R^2-1}}$ and the integral over the small green segment is bounded by $\color{#090}{2\epsilon\frac{(2\pi-\log(\epsilon))^3}{1-\epsilon^2}}$, both of which vanish as $R\to\infty$ and $\epsilon\to0$,
$$
\begin{align}
\int_\gamma\frac{\log(z)^3}{z^2+1}\,\mathrm{d}z
&=\int_0^\infty\frac{\log(x)^3-(\log(x)+2\pi i)^3}{x^2+1}\,\mathrm{d}x\\
&=\int_0^\infty\frac{-6\pi i\log(x)^2+12\pi^2\log(x)+8\pi^3i}{x^2+1}\,\mathrm{d}x\\
&=2\pi i\left[\vphantom{\left(\frac{\pi i}2\right)^3}\right.\underbrace{\left(\frac{\pi i}2\right)^3\left(\frac1{2i}\right)}_{\text{residue at }i}+\underbrace{\left(\frac{3\pi i}2\right)^3\left(-\frac1{2i}\right)}_{\text{residue at }-i}\left.\vphantom{\left(\frac{\pi i}2\right)^3}\right]\\
&=\frac{13\pi^4i}4
\end{align}
$$
Thus,
$$
-6\pi i\int_0^\infty\frac{\log(x)^2}{x^2+1}\,\mathrm{d}x
+12\pi^2\int_0^\infty\frac{\log(x)}{x^2+1}\,\mathrm{d}x
=-\frac{3\pi^4i}4
$$
which means
$$
\int_0^\infty\frac{\log(x)^2}{x^2+1}\,\mathrm{d}x=\frac{\pi^3}8
$$
and
$$
\int_0^\infty\frac{\log(x)}{x^2+1}\,\mathrm{d}x=0
$$