This response may be unnecessary after M.N.C.E.'s nice answer, but for what it's worth here's an alternative approach.
Let's start by transforming the integral via the following Euler substitution:
$$\sqrt{1-x^2}=1-xt.$$
Solving for the variable $t$ as a function of $x$ and vice versa, we find,
$$t=\frac{1-\sqrt{1-x^2}}{x},$$
and
$$x=\frac{2t}{1+t^2}.$$
The advantage of this substitution over the one suggested in the question statement is that it quickly converts the integrand to a product of a rational function and the logarithm (squared) of a rational function. A single integration by parts can then reduce the power of the logarithm to $1$:
$$\begin{align}
\mathcal{I}
&=\int_{0}^{1}\ln^2{\left(\sqrt{1+x}-\sqrt{1-x}\right)}\,\mathrm{d}x\\
&=\int_{0}^{1}\ln^2{\left(\frac{|1+t|}{\sqrt{1+t^2}}-\frac{|1-t|}{\sqrt{1+t^2}}\right)}\cdot\frac{2(1-t^2)}{(1+t^2)^2}\,\mathrm{d}t\\
&=2\int_{0}^{1}\frac{(1-t^2)}{(1+t^2)^2}\ln^2{\left(\frac{1+t-1+t}{\sqrt{1+t^2}}\right)}\,\mathrm{d}t\\
&=2\int_{0}^{1}\frac{(1-t^2)}{(1+t^2)^2}\ln^2{\left(\frac{2t}{\sqrt{1+t^2}}\right)}\,\mathrm{d}t\\
&=\frac12\int_{0}^{1}\frac{(1-t^2)}{(1+t^2)^2}\ln^2{\left(\frac{4t^2}{1+t^2}\right)}\,\mathrm{d}t\\
&=\left[\frac{t}{1+t^2}\ln^2{\left(\frac{4t^2}{1+t^2}\right)}\right]_{0}^{1}-\frac12\int_{0}^{1}\frac{t}{1+t^2}\cdot\frac{4\ln{\left(\frac{4t^2}{1+t^2}\right)}}{t(1+t^2)}\,\mathrm{d}t\\
&=\frac{\ln^2{(2)}}{4}-2\int_{0}^{1}\frac{\ln{\left(\frac{4t^2}{1+t^2}\right)}}{(1+t^2)^2}\,\mathrm{d}t.\\
\end{align}$$
The last integral of course can be systematically evaluated in terms of dilogarithms.