\begin{align}\boxed{I=\int_0^1 \ln^2(1+x^2+x^4)dx \\
= 32-12\ln 3-4\ln 2\ln 3-4 \pi \sqrt 3 +3\ln^2 3+2\pi \sqrt 3\ln 2+\frac{7\pi \ln 3}{\sqrt 3}\\
-8\sqrt 3 \operatorname{Ti}_2\left(\sqrt 3\right)+3\operatorname{Li}_2\left(-\frac13\right)+\operatorname{Li}_2\left(-3\right)-\operatorname{Li}_2\left(-8\right)}\end{align}
Where $\operatorname{Ti}_{2}(x)$ is the inverse tangent integral and $\operatorname{Li}_2(x)$ is the Dilogarithm.
To show this result we will start by using $(a+b)^2=2a^2+2b^2-(a-b)^2$ in order to rewrite the integral as:
$$I=2\int_0^1 \ln^2(1+x+x^2)dx+2\int_0^1 \ln^2(1-x+x^2)dx
-\int_0^1\ln^2\left(\frac{1+x+x^2}{1-x+x^2}\right)dx$$
$$\overset{x\to \frac{1-x}{1+x}}=\color{blue}{4\int_0^1 \frac{\ln^2\left(\frac{3+x^2}{(1+x)^2}\right)}{(1+x)^2}dx+4\int_0^1 \frac{\ln^2\left(\frac{1+3x^2}{(1+x)^2}\right)}{(1+x)^2}dx}\color{red}{-2\int_0^1 \frac{\ln^2\left(\frac{3+x^2}{1+3x^2}\right)}{(1+x)^2}dx}$$
$$\overset{x\to \frac{1}{x}}=\color{blue}{4\int_0^\infty \frac{\ln^2\left(\frac{3+x^2}{(1+x)^2}\right)}{(1+x)^2}dx}\color{red}{-\int_0^\infty \frac{\ln^2\left(\frac{3+x^2}{1+3x^2}\right)}{(1+x)^2}dx}$$
Now we will integrate by parts, in the same time we'll also simplify things using partial fractions.
\begin{align}\Rightarrow I=3\ln^2 3+12\underbrace{\int_0^\infty \frac{\ln(3+x^2)}{3+x^2}dx}_{=I_1}+4\underbrace{\int_0^\infty \frac{\ln(1+3x^2)}{3+x^2}dx}_{=I_2}-32\underbrace{\int_0^\infty \frac{\ln(1+x)}{3+x^2}dx}_{=I_3(1)}\\
-16\underbrace{\int_0^\infty \frac{\ln(3+x^2)}{(1+x)^2}dx}_{=I_4}+32\underbrace{\int_0^\infty \frac{\ln(1+x)}{(1+x)^2}dx}_{=I_5} -24\underbrace{\int_0^\infty \frac{\ln(3+x^2)}{(3+x^2)(1+x)}dx}_{=I_6(1)}\\+8\underbrace{\int_0^\infty \frac{\ln(1+3x^2)}{(3+x^2)(1+x)}dx}_{=I_7(3)}+32\underbrace{\int_0^\infty \frac{\ln(1+x)}{(3+x^2)(1+x)}dx}_{=I_8(1)}\end{align}
And all that's left to do is to evaluate each integral in order to obtain the closed form.
We obtain immediately, using $\int_0^\infty \frac{\ln(a+x^2)}{b+x^2}dx=\frac{\pi}{b}\ln(a+b)$ that:
$$I_1=\int_0^\infty \frac{\ln(3+x^2)}{(3+x^2)}dx=\frac{\pi \ln 2}{\sqrt 3}+\frac{\pi \ln 3}{2\sqrt 3}$$
$$I_2=\int_0^\infty \frac{\ln(1+3x^2)}{(3+x^2)}dx=\frac{2\pi \ln 2}{\sqrt 3}$$
$$I_3(t)=\int_0^\infty \frac{\ln(1+tx)}{3+x^2}dx\Rightarrow I_3'(t)=\int_0^\infty\frac{x}{(1+tx)(3+x^2)} dx$$
$$=\frac{3\pi}{2\sqrt 3}\frac{t}{1+3t^2}-\frac{\ln 3}{2}\frac{1}{1+3t^2}-\frac{\ln t}{1+3t^2}$$
$$I_3(1)=\int_0^1 I_3'(t)dt=\frac{\pi \ln 2}{2\sqrt 3}-\frac{\pi \ln 3}{6\sqrt 3}+\frac{\operatorname{Ti}_2(\sqrt 3)}{\sqrt 3}$$
Where the inverse tangent integral appears after integrating by parts the last term as:
$$\small \int_0^1 \frac{\ln t}{1+3t^2}dt\overset{IBP}=-\frac{1}{\sqrt 3}\int_0^1\frac{\arctan(\sqrt 3t)}{t}dt\overset{\sqrt 3 t=x}=-\frac{1}{\sqrt 3}\int_0^\sqrt 3\frac{\arctan x}{x}dx=-\frac{\operatorname{Ti}_2(\sqrt 3)}{\sqrt 3}$$
$$I_4=\int_0^\infty \frac{\ln(3+x^2)}{(1+x)^2}dx\overset{IBP}=\frac{3\pi}{4\sqrt 3}+\frac34\ln 3$$
$$I_5=\int_0^\infty\frac{\ln(1+x)}{(1+x)^2}dx\overset{IBP}=1$$
$$I_6(t)=\int_0^\infty \frac{\ln(3+tx^2)}{(3+x^2)(1+x)}dx\Rightarrow I_6'(t)=\int_0^\infty \frac{x^2}{(3+tx^2)(3+x^2)(1+x)}dx$$
$$=\frac{1}{8}\frac{\ln \left(\frac3t\right)}{3+t}+\frac{\pi}{8\sqrt 3}\frac{\sqrt t}{3+t}-\frac{1}{8}\frac{\ln t}{1- t}-\frac{\pi}{8\sqrt 3}\frac{1}{1+\sqrt t}$$
$$\small I_6(1)=\int_0^1I_6'(t)dt+\underbrace{\frac{\pi\ln 3}{8\sqrt 3}+\frac{\ln^2 3}{8}}_{=I_6(0)}=\frac{\ln 2 \ln 3}{4}-\frac18\operatorname{Li}_2\left(-\frac13\right)-\frac{\pi^2}{48}+\frac{\pi\ln 2}{4\sqrt 3}+\frac{\pi\ln 3}{8\sqrt 3}$$
Also that Dilogarithm comes from the first term, since:
$$ \int_0^1 \frac{\ln\left(\frac{3}{t}\right)}{3+t}dt\overset{\frac{t}{3}=x}=-\int_0^\frac13 \frac{\ln t}{1+t}dt\overset{IBP}=2\ln 2\ln 3-\ln^2 3-\operatorname{Li}_2\left(-\frac13\right)$$
$$I_7(t)=\int_0^\infty \frac{\ln(1+tx^2)}{(3+x^2)(1+x)}dx\Rightarrow I_7'(t)=\int_0^\infty \frac{x^2}{(1+tx^2)(3+x^2)(1+x)}dx$$
$$=\frac{\pi}{8}\frac{\sqrt t}{1+t}+\frac{3\pi \sqrt t -\pi\sqrt 3}{8}\frac{1}{1-3t}-\frac18\frac{\ln t}{1+t}-\frac38\frac{\ln (3t)}{1-3t}$$
$$I_7(3)=\int_0^3I_7'(t)dt=\frac{\pi \ln 2}{2\sqrt 3}-\frac{\pi^2}{16}-\frac{\ln 2\ln 3}{4}-\frac{\operatorname{Li}_2(-3)}{8}-\frac18\operatorname{Li}_2(-8)$$
It's perhaps worth to mention here the last integral:
$$\small \int_0^1 \frac{\ln(3t)}{1-3t}dt\overset{3t=x}=\frac13\int_0^9\frac{\ln x}{1-x}dx=\frac13\operatorname{Li}_2(1-x)\bigg|_0^9=\frac13\operatorname{Li}_2(-8)-\frac{\pi^2}{18}$$
$$I_8(t)=\int_0^\infty \frac{\ln(1+tx)}{(3+x^2)(1+x)}dx\Rightarrow I_8'(t)=\int_0^\infty\frac{x}{(1+tx)(3+x^2)(1+x)} dx$$
$$=\frac14\frac{\ln t}{1-t}+\frac{3\pi}{8\sqrt 3}\frac{1+t}{1+3x^2}+\frac{\ln 3}{8}\frac{3t-1}{1+3t^2}-\frac14\frac{\ln t}{1+3t^2}+\frac34\frac{t\ln t}{1+3t^2}$$
$$I_8(1)=\int_0^1 I_8'(t)dt=\frac{\pi\ln 2}{8\sqrt 3}+\frac{\ln 2\ln 3}{8}-\frac{\pi \ln 3}{24\sqrt 3}+\frac{\operatorname{Ti}_2(\sqrt 3)}{4\sqrt 3}+\frac{\operatorname{Li}_2(-3)}{16}$$
