I learnt that $$\int_{0}^{\frac{\pi}{2}} x^2 \ln^2 \cos x \ dx = \frac{11 \pi^5}{1440} + \frac{\pi^3}{24} \ln^2 2 + \frac{\pi}{2}\zeta(3) \ln 2$$ from Sangchul Lee's answer on How to evaluate $I=\int_0^{\pi/2}x^2\ln(\sin x)\ln(\cos x)\ dx$
I did some other calculations, and it appears that $$I=\int_{0}^{2\pi}x^2\ln^2(1-\cos x)~dx = \frac{48\pi\zeta(3)\ln2+8\pi^3\ln^22}{3}+\frac{52\pi^5}{45}.$$ However, I am not sure how to verify the result. What method should I use to calculate $I$?
After spending a lot of time I've reached the answer (not without help of "MathStackExchangians"). I'll continue the derivation by Larry starting from $I_2$.
I will use the following integral:
$$\int_0^\pi x^2\cos(2kx)~dx=\frac\pi{2k^2}$$
We have
$$\small\begin{align} I_2 &= 32\int_{0}^{\pi}x^2\ln^2(\sin x)~dx\\ &= 32\int_{0}^{\pi}x^2\left(\ln(2)+\sum_{n=1}^{\infty}\frac{\cos (2nx)}{n}\right)^2~dx\\ &= 32\int_{0}^{\pi}x^2\ln^2(2)~dx+64\ln(2)\int_{0}^{\pi}x^2\sum_{n=1}^{\infty}\frac{\cos (2nx)}{n}~dx+32\int_{0}^{\pi}x^2\left(\sum_{n=1}^{\infty}\frac{\cos (2nx)}{n}\right)^2~dx\\ &=\frac{32}{3}\pi^3\ln^2(2)+\sum_{n=1}^{\infty}\frac{64\ln(2)}{n}\int_{0}^{\pi}x^2\cos(2nx)~dx+32\underbrace{\int_{0}^{\pi}x^2\left(\sum_{n=1}^{\infty}\frac{\cos (2nx)}{n}\right)^2~dx}_{J} \end{align}$$
$$\small\begin{align} J &= \int_{0}^{\pi}x^2\left(\sum_{n=1}^{\infty}\frac{\cos (2nx)}{n}\right)^2~dx\\ &=\int_{0}^{\pi}x^2\left(\sum_{n=1}^{\infty}\frac{\cos^2 (2nx)}{n^2}+\sum_{m,n=1;m\neq n}^{\infty}\frac{\cos (2mx)\cos (2nx)}{mn}\right)~dx\\ &=\sum_{n=1}^{\infty}\frac1{n^2}\int_{0}^{\pi}x^2\cos^2 (2nx)~dx+\sum_{m,n=1;m\neq n}^{\infty}\frac1{mn}\int_{0}^{\pi}x^2\cos (2mx)\cos (2nx)~dx\\ &=\sum_{n=1}^{\infty}\frac1{2n^2}\int_{0}^{\pi}x^2(1+\cos (4nx))~dx+\sum_{m,n=1;m\neq n}^{\infty}\frac1{2mn}\int_{0}^{\pi}x^2(\cos (2(m+n)x)+\cos (2(m-n)x))~dx\\ &=\sum_{n=1}^{\infty}\frac1{2n^2}\left(\int_{0}^{\pi}x^2~dx+\int_{0}^{\pi}x^2\cos (4nx)~dx\right)+\sum_{m,n=1;m\neq n}^{\infty}\frac1{2mn}\left(\int_{0}^{\pi}x^2\cos (2(m+n)x)~dx+\int_{0}^{\pi}x^2\cos (2(m-n)x)~dx\right)\\ &=\sum_{n=1}^{\infty}\frac1{2n^2}\left(\frac{\pi^3}3+\frac\pi{2(2n)^2}\right)+\sum_{m,n=1;m\neq n}^{\infty}\frac1{2mn}\left(\frac\pi{2(m+n)^2}+\frac\pi{2(m-n)^2}\right)\\ &=\frac{\pi^3}6\sum_{n=1}^{\infty}\frac1{n^2}+\frac\pi{16}\sum_{n=1}^{\infty}\frac1{n^4}+\frac\pi2\sum_{m,n=1;m\neq n}^{\infty}\frac{m^2+n^2}{mn(m^2-n^2)^2}\\ &=\frac{\pi^3}6\frac{\pi^2}6+\frac{\pi}{16}\frac{\pi^4}{90}+\frac\pi2\frac{11\pi^4}{720}=\frac{13\pi^5}{360} \end{align} $$
The last sum is evaluated (my thanks to Robert Z and Zvi) in this question
Finally we have
$$\small\begin{align} I_2 &= \frac{32}{3}\pi^3\ln^2(2)+\sum_{n=1}^{\infty}\frac{64\ln(2)}{n}\int_{0}^{\pi}x^2\cos(2nx)~dx+32\frac{13\pi^5}{360}\\ &= \frac{32}{3}\pi^3\ln^2(2)+\sum_{n=1}^{\infty}\frac{64\ln(2)}{n}\frac\pi{2n^2}+\frac{52\pi^5}{45}\\ &=\frac{32}{3}\pi^3\ln^2(2)+32\ln(2)\pi\sum_{n=1}^{\infty}\frac1{n^3}+\frac{52\pi^5}{45}\\ &= \pi^3\ln^2(2)+32\pi\ln(2)\zeta(3)+\frac{52\pi^5}{45} \end{align}$$
And thus
$$\begin{align} I&=I_1+I_2\\ &= -8\pi^3\ln^2(2)-16\pi\ln(2)\zeta(3)+\frac{32}{3}\pi^3\ln^2(2)+32\pi\ln(2)\zeta(3)+\frac{52\pi^5}{45}\\ &= 16\pi\zeta(3)\ln(2)+\frac{8\pi^3\ln^2(2)}{3}+\frac{52\pi^5}{45} \end{align}$$
Attempt:
By using the trigonometric identity
$$1-\cos(x)=2\sin^2\left(\frac x2\right)$$
The given integral becomes
$$\small\begin{align} I = \int_{0}^{2\pi}x^2\ln^2 (1-\cos x)~dx &= \int_{0}^{2\pi}x^2 \ln^2\left(2\sin^2\left(\frac x2\right)\right)~dx\\ &=\int_{0}^{2\pi}x^2\left(\ln(2)+2\ln\left(\sin \frac{x}{2}\right)\right)^2dx\\ &=\int_{0}^{2\pi}x^2 \left(\ln^2(2)+4\ln(2)\ln\left(\sin \frac{x}{2}\right)+4\ln^2\left(\sin \frac{x}{2}\right)\right)dx\\ &=\frac{8\pi^3}{3}\ln^2(2)+32\ln(2)\int_0^{\pi}x^2\ln(\sin x)~dx+4\int_{0}^{2\pi}x^2\ln^2\left(\sin \frac{x}{2}\right)~dx \end{align}$$
where within the second integral the substitution $x=\frac x2$ was used. Now use the Fourier series expansion
$$\ln(\sin x)=-\ln(2)-\sum_{n=1}^{\infty}\frac{\cos(2nx)}{n}$$
to further get
$$\small\begin{align} I_1 = \frac{8\pi^3}{3}\ln^2(2)+32\ln(2)\int_0^{\pi}x^2\ln(\sin x)~dx&=\frac{8\pi^3}{3}\ln^2(2)+32\ln(2)\int_0^{\pi}x^2\left[-\ln(2)-\sum_{n=1}^{\infty}\frac{\cos(2nx)}{n}\right]~dx\\ &=\frac{8\pi^3}{3}\ln^2(2)-32\ln^2(2)\int_0^{\pi}x^2~dx-\sum_{n=1}^{\infty}\frac{32\ln(2)}n\int_0^{\pi}x^2\cos(2nx)~dx\\ &=-8\pi^3\ln^2(2)-\sum_{n=1}^{\infty}\frac{32\ln(2)}n\int_0^{\pi}x^2\cos(2nx)~dx \end{align}$$ Using integration by part, we get $$\small\begin{align} I_1 &= -8\pi^3\ln^2(2)-\sum_{n=1}^{\infty}\frac{32\ln(2)}n\int_0^{\pi}x^2\cos(2nx)~dx\\ &=-8\pi^3\ln^2(2)-\sum_{n=1}^{\infty}\frac{32\ln(2)}n\left[-x^2\frac{\sin(2nx)}{2n}+\frac{2x\cos(2nx)}{4n^2}-\frac{2\sin(2nx)}{8n^3}\right]_{0}^{\pi}\\ &=-8\pi^3\ln^2(2)-\sum_{n=1}^{\infty}\frac{32\ln(2)}{n}\frac{2\pi}{4n^2}\\ &= -8\pi^3\ln^2(2)-16\pi\ln(2)\sum_{n=1}^{\infty}\frac{1}{n^3}\\ &= -8\pi^3\ln^2(2)-16\pi\ln(2)\zeta(3) \end{align}$$ Let $$I_2 = 4\int_{0}^{2\pi}x^2\ln^2\left(\sin \frac{x}{2}\right)~dx$$ Again, use $x = \frac{x}{2}$ $$\small\begin{align} I_2 &= 32\int_{0}^{\pi}x^2\ln^2(\sin x)~dx\\ &= 32\int_{0}^{\pi}x^2\left(\ln(2)+\sum_{n=1}^{\infty}\frac{\cos (2nx)}{n}\right)^2~dx\\ &= 32\int_{0}^{\pi}x^2\ln^2(2)~dx+64\ln(2)\int_{0}^{\pi}x^2\sum_{n=1}^{\infty}\frac{\cos (2nx)}{n}~dx+32\int_{0}^{\pi}x^2\left(\sum_{n=1}^{\infty}\frac{\cos (2nx)}{n}\right)^2~dx\\ &=\frac{32}{3}\pi^3\ln^2(2)+\sum_{n=1}^{\infty}\frac{64\ln(2)}{n}\int_{0}^{\pi}x^2\cos(2nx)~dx+32\int_{0}^{\pi}x^2\left(\sum_{n=1}^{\infty}\frac{\cos (2nx)}{n}\right)^2~dx\\ &=\frac{32}{3}\pi^3\ln^2(2)+32\pi\ln(2)\zeta(3)+32\int_{0}^{\pi}x^2\sum_{n=1}^{\infty}\frac{\cos(2nx)}{n}\sum_{n=1}^{\infty}\frac{\cos(2nx)}{n}~dx\\ &=\frac{32}{3}\pi^3\ln^2(2)+32\pi\ln(2)\zeta(3)+\sum_{n=1}^{\infty}\frac{32}{n}\int_{0}^{\pi}x^2\cos(2nx)\sum_{n=1}^{\infty}\frac{\cos(2nx)}{n}~dx\\ &=\frac{32}{3}\pi^3\ln^2(2)+32\pi\ln(2)\zeta(3)+\sum_{n=1}^{\infty}\frac{32}{n}\sum_{n=1}^{\infty}\frac{1}{n}\int_{0}^{\pi}x^2\cos^2(2nx)~dx\\ &=\frac{32}{3}\pi^3\ln^2(2)+32\pi\ln(2)\zeta(3)+\sum_{n=1}^{\infty}\frac{32}{n^2}\left[\frac{x^2\sin(4nx)}{8n}-\frac{\sin(4nx)}{64n^3}+\frac{x\cos(4nx)}{16n^2}+\frac{x^3}{6}\right]_{0}^{\pi}\tag{a}\\ &=\frac{32}{3}\pi^3\ln^2(2)+32\pi\ln(2)\zeta(3)+\sum_{n=1}^{\infty}\frac{32}{n^2}\left(\frac{\pi}{16n^2}+\frac{\pi^3}{6}\right)\\ &=\frac{32}{3}\pi^3\ln^2(2)+32\pi\ln(2)\zeta(3)+\sum_{n=1}^{\infty}\frac{2\pi}{n^4}+\frac{16\pi^3}{3n^2}\\ &=\frac{32}{3}\pi^3\ln^2(2)+32\pi\ln(2)\zeta(3)+\frac{\pi^5}{45}+\frac{8\pi^5}{9}\\ &=\frac{32}{3}\pi^3\ln^2(2)+32\pi\ln(2)\zeta(3)+\frac{41\pi^5}{45} \end{align}$$ Note that $$\begin{align} I&=I_1+I_2\\ &= -8\pi^3\ln^2(2)-16\pi\ln(2)\zeta(3)+\frac{32}{3}\pi^3\ln^2(2)+32\pi\ln(2)\zeta(3)+\frac{41\pi^5}{45}\\ &= \frac{48\pi\zeta(3)\ln(2)+8\pi^3\ln^2(2)}{3}+\frac{41\pi^5}{45} \end{align}$$ However, the last term should be $\frac{52\pi^5}{45}$. I think I did something wrong on step (a).
