Presented below is a self-contained evaluation. Let $J=\int_0^1 \frac{\ln^2(1+x)}{1+x^2}dx$
\begin{align}
&\int_0^1 \frac{\ln^2(1-x)}{1+x^2}dx
\overset{x\to\frac{1-x}{1+x}}= J
-2 \int_0^1 \frac{\ln x\ln(1+x)}{1+x^2}dx-2G\ln2+\frac{\pi^3}{16}\\
&\int_0^\infty \frac{\ln^2(1+x)}{1+x^2}dx
\overset{(0,1)+\overset{x\to 1/x}{(1,\infty)}} = 2J -2 \int_0^1 \frac{\ln x\ln(1+x)}{1+x^2}dx+\frac{\pi^3}{16}
\end{align}
Eliminate $J$ to get
$$\int_0^1 \frac{\ln x\ln(1+x)}{1+x^2}dx
=\frac12\int_0^\infty \frac{\ln^2(1+x)}{1+x^2}dx-K-2G\ln2+\frac{\pi^3}{32}
\tag1$$
where $K=\int_0^1 \frac{\ln^2(1-x)}{1+x^2}dx =2\text{Im}\>\text{Li}_3\left(\frac{1+i}2\right)
$. Also
$$\int_0^1 \frac{\ln x\ln(1-x)}{1+x^2}dx
=\frac12 K-\frac12\int_0^1\frac{\ln\frac x{1-x}}{1+x^2}dx +\frac12\int_0^1\frac{\ln^2x}{1+x^2}dx\tag2
$$
Then, (1) + (2)
$$
\int_0^1 \frac{\ln x\ln(1-x^2)}{1+x^2}dx
=-2\text{Im}\>\text{Li}_3\left(\frac{1+i}2\right)-2G\ln2+\frac{\pi^3}{16} +\frac12 A \tag3$$
where
\begin{align}
A =& \int_0^1 \underset{t=1-x}{\frac{\ln^2(1-x)}{1+x^2}}dx- \int_0^1\underset{t=\frac x{1-x}}{\frac{\ln\frac x{1-x}}{1+x^2}}dx +\underset{t=1+x}{
\int_0^\infty \frac{\ln^2(1+x)}{1+x^2}}dx \\
=& \int_0^\infty {\frac{4t \ln^2 t}{4+t^4}}dt
\overset{t^2=2u}= \frac14 \int_0^\infty \frac{\ln^2(2u)}{1+u^2}du
= \frac{\pi}{8}\ln^22+\frac{\pi^3}{32}
\end{align}
Plug into (3) to obtain
$$
\int_0^1 \frac{\ln x\ln(1-x^2)}{1+x^2}dx =-2\text{Im}\>\text{Li}_3\left(\frac{1+i}2\right) -2G\ln2
+\frac{\pi}{16}\ln^22+\frac{5\pi^3}{64}$$