How to prove that $\displaystyle \int_0^{\infty} \dfrac{\arctan(x)\sqrt{\sqrt{1+x^2}-1}}{x\sqrt{1+x^2}}\,dx= \frac{1}{4}\pi^2\sqrt{2}-\sqrt{2}\ln^2\left(1+\sqrt{2}\right)$ ?
It's very difficult and I have no idea.
How to prove that $\displaystyle \int_0^{\infty} \dfrac{\arctan(x)\sqrt{\sqrt{1+x^2}-1}}{x\sqrt{1+x^2}}\,dx= \frac{1}{4}\pi^2\sqrt{2}-\sqrt{2}\ln^2\left(1+\sqrt{2}\right)$ ?
It's very difficult and I have no idea.
This can be reduced to a class of integral related to polylogarithm, and is related to Legendre-chi function $\chi_2$. The formula $$\int_{0}^{\frac{\pi}{2}} \arctan(r \sin\theta) \, d\theta = 2 \chi_{2} \left( \frac{\sqrt{1+r^{2}} - 1}{r} \right) $$ gives $$\tag{1}\int_{0}^{\frac{\pi}{2}} \arctan( \sin\theta) \ d\theta = 2 \chi_{2} \left( \sqrt{2}-1 \right) = \frac{\pi^2}{8} - \frac{1}{2}\log^2(\sqrt{2}+1)$$ where we used a famous special value of $\chi_2$. For a "quick introduction" about $\chi_2$, see Sangchul Lee's here.
Return to your problem, denote your integral by $I$, note that $$\int \frac{\sqrt{\sqrt{1+x^2}-1}}{x\sqrt{1+x^2}} dx = \sqrt 2 \arctan \left[ {\frac{{\sqrt {\sqrt {1 + {x^2}} - 1} }}{{\sqrt 2 }}} \right]$$ so integration by parts gives $$I = \frac{\sqrt{2}\pi^2}{4}-\sqrt{2}\int_0^\infty {\arctan \left[ {\frac{{\sqrt {\sqrt {1 + {x^2}} - 1} }}{{\sqrt 2 }}} \right]} \frac{{dx}}{{1 + {x^2}}}$$ making $x\mapsto \tan x$ into the last integral gives $$I = \frac{\sqrt{2}\pi^2}{4}-\sqrt{2}\underbrace{\int_0^{\pi/2} {\arctan \left( {\frac{{\sin \frac{x}{2}}}{{\sqrt {\cos } x}}} \right)dx}}_J $$
Denote $$F(r)=\int_0^{\pi/2} {\arctan \left( {\frac{{a\sin \frac{x}{2}}}{{\sqrt {\cos } x}}} \right)} dx$$ Then compute $F'(r)$ and do an indefinite integral gives $$\begin{aligned} F'(r) &= \int_0^{\pi /2} {\frac{{\sqrt {\cos x} \sin (\frac{x}{2})}}{{\cos x + {a^2}\sin {{(\frac{x}{2})}^2}}}} dx \\ & = - \left[ {\frac{{2a\arctan \left[ {\frac{{a\cos (\frac{x}{2})}}{{\sqrt {1 - {a^2}} \sqrt {\cos x} }}} \right]}}{{(2 - {a^2})\sqrt {1 - {a^2}} }} + \frac{{2\sqrt 2 }}{{2 - {a^2}}}\ln \left[ {\sqrt 2 \cos (\frac{x}{2}) + \sqrt {\cos x} } \right]} \right]_0^{\pi /2} \\ &= - \frac{{\pi a}}{{(2 - {a^2})\sqrt {1 - {a^2}} }} + \frac{{2a\arctan \left[ {\frac{a}{{\sqrt {1 - {a^2}} }}} \right]}}{{(2 - {a^2})\sqrt {1 - {a^2}} }} + \frac{{2\sqrt 2 \ln (\sqrt 2 + 1)}}{{2 - {a^2}}}\end{aligned}$$
Then to get $J$, we integrate the whole expression with respect to $a$ from $0$ to $1$, only the second integral deserves attention, applying $a\mapsto \sin x$: $$\begin{aligned}\int_0^1 {\frac{{\arctan \left[ {\frac{a}{{\sqrt {1 - {a^2}} }}} \right]}}{{(2 - {a^2})\sqrt {1 - {a^2}} }}da} &= \int_0^{\pi /2} {\frac{{x\cos x}}{{2 - {{\sin }^2}x}}dx} \\&=- \int_0^{\frac{\pi }{2}} {xd\left[ {\arctan (\cos x)} \right]} \\&= \int_0^{\frac{\pi }{2}} {\arctan (\cos x)dx} \end{aligned}$$ Then use $(1)$. Combining all above evaluation should give $$J=\int_0^{\pi/2} {\arctan \left( {\frac{{\sin \frac{x}{2}}}{{\sqrt {\cos } x}}} \right)}dx=\ln^2(\sqrt{2}+1)$$ this proves your result.
Probably there is some smart way to reduce $J$ directly into $(1)$, I hope someone can point on how.
Let $x = \tan u \implies dx = \sec^2u\ du$
$$\begin{align}\int_0^{\infty} \dfrac{\arctan(x)\sqrt{\sqrt{1+x^2}-1}}{x\sqrt{1+x^2}}\,dx =& \int_0^{\pi/2} \dfrac{\arctan(\tan u)\sqrt{\sec u-1}\sec u}{\tan u }\,du &\\=& \int_0^{\pi/2} \dfrac{u\sqrt{\sec u-1}}{\sin u }\,du&\\=& \int_0^{\pi/2} \dfrac{u\sqrt{1-\cos u}}{\sin u \sqrt{\cos u}}\,du&\\=& \sqrt{2}\int_0^{\pi/2} \dfrac{u\sin u/2}{\sin u \sqrt{\cos u}}\,du &\\=& \dfrac{1}{\sqrt{2}}\int_0^{\pi/2} \dfrac{u}{\cos (u/2) \sqrt{\cos u}}\,du&\\=& \dfrac{1}{\sqrt{2}}\left(\left[2u \sin^{-1}\left(\tan \dfrac u2\right)\right]_0^{\pi/2} - 2\int_0^{\pi/2}\sin^{-1}\left(\tan \dfrac u2\right)\, du\right)&\\=& \dfrac{\pi^2}{2\sqrt{2}} - \dfrac{2}{\sqrt{2}}\int_0^{\pi/2}\sin^{-1}\left(\tan \dfrac u2\right)\, du \end{align} $$
Let $z = u/2 \implies 2dz = du$ $$\begin{align}\int_0^{\pi/2}\sin^{-1}\left(\tan \dfrac u2\right)\, du&=& 2\int_0^{\pi/4}\sin^{-1}(\tan z )\, dz\end{align}$$
Let $t = \tan z$
$$\begin{align} 2\int_0^{\pi/4}\sin^{-1}(\tan z )\, dz=& 2\int_0^1\dfrac{\sin^{-1} t}{t^2 + 1} dt&\\=& 2[\tan^{-1} x \sin^{-1} x]_0^1 - 2\int_0^1 \dfrac{\tan^{-1}(t)}{\sqrt{1-t^2}}\, dt&\\=& 2\dfrac{\pi^2}{8} - 2\int_0^1 \dfrac{\tan^{-1}(t)}{\sqrt{1-t^2}}\, dt\end{align}$$
The integral $$\int_0^1 \dfrac{\tan^{-1}(t)}{\sqrt{1-t^2}}\, dt = -\frac{\ln^{2}(1+\sqrt{2})}{2} + \frac{\pi^{2}}{8} \tag 1$$
So,
$$\int_0^{\pi/2}\sin^{-1}\left(\tan \dfrac u2\right)\, du = \ln^2(1+ \sqrt{2})$$
The final answer is then
$$\dfrac{\pi^2}{2\sqrt{2}} - \sqrt{2}\ln^2(1+\sqrt{2})$$.
For the proof of $(1)$ see here.
$\begin{align}I&=\int_0^{\infty} \dfrac{\arctan(x)\sqrt{\sqrt{1+x^2}-1}}{x\sqrt{1+x^2}}\,dx \end{align}$
Perform the change of variable $x=\dfrac{2y}{1-y^2}$,
$\begin{align}I&=\sqrt{2}\int_0^{1} \dfrac{\arctan\left(\dfrac{2y}{1-y^2}\right)}{\sqrt{1-y^2}}\,dy\\ &=2\sqrt{2}\int_0^{1} \dfrac{\arctan y}{\sqrt{1-y^2}}\,dy\\ \end{align}$
If one assume that,
$\displaystyle \int_0^{1} \dfrac{\arctan y}{\sqrt{1-y^2}}\,dy=\frac{1}{8}\pi^2-\frac{1}{2}\ln^2(\sqrt{2}+1)$
therefore,
$\displaystyle \boxed{I=\frac{1}{4}\pi^2\sqrt{2}-\sqrt{2}\ln^2(\sqrt{2}+1)}$
(for a proof of this, see: Evaluating a sum involving binomial coefficient in denominator )
Through the substitution $x=\tan\theta$ the integral is converted into $$ \int_{0}^{\pi/2}\frac{\theta}{\sin\theta}\sqrt{\frac{1}{\cos\theta}-1}\,d\theta=\int_{0}^{\pi/2}\frac{\theta}{\cos\frac{\theta}{2}\sqrt{2\cos\theta}}\,d\theta$$ and by integration by parts the RHS equals $$ \frac{\pi^2}{2\sqrt{2}}-\sqrt{2}\int_{0}^{\pi/2}\arctan\left(\frac{\sin\frac{\theta}{2}}{\sqrt{\cos\theta}}\right)\,d\theta $$ where $$\begin{eqnarray*}\int_{0}^{\pi/2}\arctan\left(\frac{\sin\frac{\theta}{2}}{\sqrt{\cos\theta}}\right)\,d\theta &=& \int_{0}^{\pi/2}\arctan\sqrt{\frac{1-\cos\theta}{2\cos\theta}}\,d\theta\\&=&\int_{0}^{\pi/2}\arctan\sqrt{\frac{1-\sin\theta}{2\sin\theta}}\,d\theta\\&=&\int_{0}^{1}\arctan\sqrt{\frac{1-x}{2x}}\,\frac{dx}{\sqrt{1-x^2}}\\&=&\int_{0}^{+\infty}\frac{\arctan\sqrt{t}}{(1+2t)\sqrt{t(1+t)}}\,dt\\&=&\int_{0}^{+\infty}\frac{2\arctan u}{(1+2u^2)\sqrt{1+u^2}}\,du\end{eqnarray*}$$ by Feynman's trick, is related to the integral $$ \int_{0}^{1}\frac{K(\sqrt{x})}{\sqrt{2-x}}\,dx = \frac{\pi^2}{4}-\log^2(1+\sqrt{2}),$$ which can be computed through Fourier-Legendre series expansion (see here, page 19) and the dilogarithm reflection formula.