The integral: $$\int_{0}^{\pi/2} \frac{\sin x \cos^5 x}{(1-2\sin^2x\cos^2x)^2}dx$$ has been encountered today while solving a longlish problem at MSE. The question here is: How would one evaluate it?
Addendum
For an interesting use of this integral see my Answer to: Evaluate: $S=\sum_{j=1}^{\infty}\prod_{k=1}^{j}\frac{2k}{j+k+1}$
Define $u=2\cos^{2}{(x)}-1$, then
$$ \begin{aligned} \int_{0}^{\frac{\pi}{2}}{\frac{\sin{(x)}\cos^{5}{(x)}}{(1-2\sin^{2}{(x)}\cos^{2}{(x)})^{2}}dx}&=\frac{1}{4}\left(\int_{-1}^{1}{\frac{du}{u^{2}+1}}+\int_{-1}^{1}{\frac{2u\ du}{(u^{2}+1)^{2}}}\right)\\ \\ &=\frac{\pi}{8} \end{aligned} $$
Let $u= \cos x$, $du = - \sin x \ dx$:
$$\int_{0}^{\pi/2} \frac{\sin x \cos^5 x}{(1-2\sin^2x\cos^2x)^2}dx$$ $$= -\int_{1}^{0} \frac{du \ (u^5)}{(1-2(1-u^2)u^2)^2}$$ $$= \int_{0}^{1} \frac{u^5}{(1-2u^2+2u^4)^2} \ du$$
Then substitute again: let $v = u^2, dv = 2u\ du$:
$$= \frac{1}{2} \int_{0}^{1} \frac{v^2 \ dv}{(1-2v+2v^2)^2}$$
and do (not really) partial fractions:
$$ \frac{v^2 }{(1-2v+2v^2)^2} = \frac{a}{(1-2v+2v^2)} + \frac{b}{(1-2v+2v^2)^2}$$ $$ v^2= a(1-2v+2v^2) + b$$ $$a = \frac{1}{2} \Rightarrow v^2 = \frac{1}{2}-v+v^2+b$$ $$b = v - \frac{1}{2}$$
So we have:
$$\frac{1}{4} \int_{0}^{1} \frac{dv}{(1-2v+2v^2)} + \frac{1}{2} \int_{0}^{1} \frac{v \ dv}{(1-2v+2v^2)^2} - \frac{1}{2} \int_{0}^{1} \frac{\ dv}{(1-2v+2v^2)^2}$$
For the first integral, complete the square which resolves to the standard $\arctan$ integral. For the second integral, let $w = 2v - 1$ (AoPS), and for the third integral, complete the square, then substitute $w = \frac{\tan v}{\sqrt 2}$ where you can use $\tan^2 w + 1 = \sec^2 w$.
$$I=\int_{0}^{\pi/2} \frac{\sin x \cos^5x}{(1-2\sin^2 x \cos^2 x)^2} dx$$ Next using $\int_{0}^{a} f(x) dx= \int_{0}^{a} f(a-x) dx,$ we get $$I=\int_{0}^{\pi/2} \frac{\sin^5 x \cos x}{(1-2\sin^2 x \cos^2 x)^2} dx$$ Adding the two integrals, we get So $$2I=\int_{0}^{\pi/2} \frac{\sin x \cos x(\sin^4 x+ \cos^4 x)}{(1-2\sin^2 x \cos^2 x)^2}=\int_{0}^{\pi/2} \frac{\sin x \cos x}{(1-2\sin^2 x \cos^2 x)}dx=\int_{0}^{\pi/2}\frac{1}{2} \frac{\sin 2x dx}{1+\cos^2 2x}$$ $$ \text{domain halved,}~\implies 2 I=\int_{0}^{\pi/4}\frac{2\sin 2x dx}{1+\cos^2 2x}=-\int_{1}^{0}\frac{dt}{1+t^2}=\frac{\pi}{4} \implies I=\frac{\pi} {8} $$ Lastly we have used $\cos 2x=t.$
Amazing : the tangent half angle substitution works. Let $x=2\tan^{-1}(t)$ to get $$I=-4\int\frac{ t \left(t^2-1\right)^5 \left(t^2+1\right)}{\left(t^8-4 t^6+22 t^4-4 t^2+1\right)^2}\,dt$$ Now, a non-trivial substitution $$t=\sqrt{1+\frac{2 \left(\sqrt{z+1}\right)}{z}}\implies dt=-\frac{1}{2} \sqrt{\frac{z+ 2 \left(\sqrt{z+1}\right)}{z^3(z+1)}}\,dz$$ $$I=\frac{1}{2} \int \frac{dz}{ \left(z^2+1\right)^2}=\frac{1}{4} \left(\frac{z}{z^2+1}+\tan ^{-1}(z)\right)$$ Back to $t$ $$I=\frac{t^2 \left(t^2-1\right)^2}{t^8-4 t^6+22 t^4-4 t^2+1}+\frac{1}{4} \tan ^{-1}\left(\frac{4 t^2}{\left(1-t^2\right)^2}\right)$$
Expanded as a series around $t=1$ gives for the definite integral $$\frac{\pi }{8}-\frac{1}{6} (1-t)^6+O\left((1-t)^7\right)$$
Seeing that cosine to an odd power, I would factor out one cosine to use with the derivative, then write the rest of the integrand in terms of sine: $\int_0^{\pi/2}\frac{sin(x)cos^5(x)}{1- 2cos^2(x)sin^2(x)}dx= \int_0^{\pi/2}\frac{sin(x)cos^4(x)}{1- 2cos^2(x)sin^2(x)}(cos(x)dx)= \int_0^{\pi/2} \frac{sin(x)((1- sin^2(x))^2}{1- 2(1- sin^2(x))sin^2(x)}(cos(x)dx)$.
Let u= sin(x) so that du= cos(x)dx. When x= 0, u= 0 and when $x= \pi/2$ u= 1 so the integral becomes $\int_0^1 \frac{u(1- u^2)^2}{1- 2(1- u^2)u^2}du$
Rewrite the integrand in terms of $\tan$, then substitute $\sqrt u=\tan x$:
$$\begin{align*} I &= \int_0^{\tfrac\pi2} \frac{\sin x \cos^5x}{\left(1-2\sin^2x\cos^2x\right)^2} \, dx \\ &= \int_0^{\tfrac\pi2} \frac{\tan x \sec^2x}{\left(1+\tan^4x\right)^2} \, dx \\ &= \frac12 \int_0^\infty \frac{du}{\left(1+u^2\right)^2} \end{align*}$$
Now reintroducing $\tan$ by substituting $u=\tan v$ will reduce the integral to the well-known $\displaystyle \int \cos^2v \, dv$.
To summarize, let $x = \arctan\left(\sqrt{\tan v}\right)$.
Alternatively, substitute $u=\sqrt v$ to recover a beta integral:
$$I = \int_0^\infty \frac{\sqrt v}{(1+v)^3} \, dv = \operatorname{B}\left(\frac32,\frac32\right) = \boxed{\frac\pi8}$$
