$$\int_0^{\pi/2}\sin(nx)\arctan(\sin x)\,\mathrm{d}x=\dfrac{\pi(\sqrt{2}-1)^n}{2n} \qquad(n=2m+1,m\in\mathbb{Z}_+)$$ I tried to solve it by Feynman integral method, but I got stuck at the following step:
$$\int_0^{\pi/2}\frac{\sin(nx)\sin{x}}{a^2\sin^2x+1}\mathrm{d}x$$
For this integral, I often use mathematical induction method, but it is not successful. I would appreciate your help!
Rewrite the integral as \begin{align*} I_{2m+1} &= \int_0^{\pi/2}\sin(2m+1)x\ \tan^{-1}(\sin x)\ \overset{ibp}{dx}\\ &= \frac1{4(2m+1)} \int_0^\tfrac\pi2 \frac{\cos2(m+1)x +\cos 2mx}{1+\sin^2x} \, \overset{2x\to x}{dx}\\ &= \frac1{2(2m+1)} \int_0^\pi\frac{\cos(m+1)x +\cos mx}{3-\cos x} \ dx\\ \end{align*} With $r=-(\sqrt2-1)^2$, apply the Fourier series $$\eqalign{ \frac{1-r^2}{1+r^2+2r\cos x}=1+2\sum_{k>0}(-r)^k\cos k x } $$ to the denominator, i.e. $$\frac{1}{3-\cos x}=2\sqrt2\left( 1+2\sum_{k>0} (\sqrt2-1)^{2k}\cos kx\right) $$ Then \begin{align*} I_{2m+1} &= \frac1{2\sqrt2(2m+1)} \int_0^\pi [\cos(m+1)x +\cos mx] \sum_{k>0} (\sqrt2-1)^{2k}\cos kx\ dx \\ \end{align*} Due to periodicity, only the terms of $k=m+1$ and $k=m$ survive the integration, which leads to
\begin{align*} I_{2m+1} &= \frac1{2\sqrt2(2m+1)} \int_0^\pi \left[(\sqrt2-1)^{2m+2}\cos^2(m+1)x + (\sqrt2-1)^{2m}\cos^2mx\right]dx\\ &= \frac\pi{4\sqrt2(2m+1)} \left[(\sqrt2-1)^{2m+2}+ (\sqrt2-1)^{2m}\right]= \frac{\pi(\sqrt{2}-1)^{2m+1}}{2(2m+1)} \end{align*}
Integrate by parts; the constant term vanishes since $\cos\dfrac{(2m+1)\pi}2=0$.
$$\begin{align*} I_{2m+1} &= \int_0^\tfrac\pi2 \sin((2m+1)x)\arctan(\sin x) \, dx \\ &= \frac1{2m+1} \int_0^\tfrac\pi2 \frac{\cos((2m+1)x) \cos x}{1+\sin^2x} \, dx \end{align*}$$
Now try induction. We can expand $\cos((2m+1)x)$ as a polynomial in $\cos x$ (this can be established with Euler's formula and de Moivre's theorem), then expand the integrand by long division.
For instance, with $m=1$, we can expand to
$$\frac{\cos(3x) \cos x}{1+\sin^2x} = \frac{4\cos^4x-3\cos^2x}{2-\cos^2x} = -5 - 4\cos^2x + \frac{10}{1+\sin^2x}$$
For each $m$, we'll end up having to integrate a polynomial in $\cos x$ plus a rational term, which for this case can be attacked by rewriting
$$\frac{10}{1+\sin^2x} = \frac{10\sec^2x}{\sec^2x+\tan^2x} = \frac{10\sec^2x}{1+2\tan^2x}$$
and substituting $y=\dfrac1{\sqrt2}\tan x$.
Here is a proof using complex analysis. Let $n$ be odd, and let
$$I(n)=\int_0^{\pi/2} \sin(nx)\arctan(\sin(x))$$
be your integral. We integrate by parts to get \begin{equation} \begin{split} I(n)&=-\frac1n\left[\cos(nx)\arctan(\sin(x))\right]_0^{\pi/2}+\frac1n \int_0^{\pi/2}\frac{\cos(nx)\cos(x)}{1+\sin^2(x)}dx\\ &=\frac1{4n}\int_0^{2\pi}\frac{\cos(nx)\cos(x)}{1+\sin^2(x)}dx\\ &=\frac1{4n}\int_0^{2\pi}\frac{(e^{inx}+e^{-inx})(e^{ix}+e^{-ix})}{4-(e^{ix}-e^{-ix})^2}dx\\ &=-\frac{i}{4n}\oint_{|z|=1}\frac{(z^n+z^{-n})(z+z^{-1})}{(4-(z-z^{-1})^2)z}dz\\ &=\frac{i}{4n}\oint_{|z|=1}\frac{(z^n+z^{-n})(z^2+1)}{(z^4-6z^2+1)}dz\\ \end{split} \end{equation} Note that the above integrand $f(z)$ has precisely three poles $\pm (\sqrt2-1),0$ within the unit disc. The first two poles are simple, so we may compute their residues directly. $$\text{Res}(f,\pm(\sqrt2-1))=-\frac14((\sqrt2+1)^n+(\sqrt2-1)^n)$$ To calculate the residue at $0$, we perform partial fraction decomposition.
\begin{equation} \begin{split} \frac{(z^2+1)z^{-n}}{(z^4-6z^2+1)}&=-\frac{z^{-n}}{4(\sqrt2+1-z)}-\frac{z^{-n}}{4(-\sqrt2+1-z)}+\frac{z^{-n}}{4(\sqrt2-1-z)}+\frac{z^{-n}}{4(-\sqrt2-1-z)}\\ &=-\frac{(\sqrt2-1)z^{-n}}{4(1-z(\sqrt2-1))}+\frac{(\sqrt2+1)z^{-n}}{4(1+z(\sqrt2+1))}+\frac{(\sqrt2+1)z^{-n}}{4(1-z(\sqrt2+1))}-\frac{(\sqrt2-1)z^{-n}}{4(1+z(\sqrt2-1))}\\ &=\frac12\sum_{k=0}^\infty \left[(\sqrt2+1)^{2k+1}z^{2k-n}-(\sqrt2-1)^{2k+1}z^{2k-n}\right]\\ \end{split} \end{equation} From which we may deduce that
$$\text{Res}(f,0)=[z^{-1}]\frac{(z^2+1)z^{-n}}{(z^4-6z^2+1)}=\frac12((\sqrt2+1)^n-(\sqrt2-1)^n)$$ Finally, the residue theorem tells us that
\begin{equation} \begin{split} I(n)&=-\frac{\pi}{2n}\left[\text{Res}(f,\sqrt2+1)+\text{Res}(f,-\sqrt2-1)+\text{Res}(f,0)\right]\\ &=\frac{\pi(\sqrt2-1)^n}{2n} \end{split} \end{equation} as desired.
Two related integral formulas are $$I(m,r) = \int_{0}^{\pi/2} \sin \left((2m+1)x \right) \arctan \left(\frac{2r \sin x}{1-r^{2}} \right) \, \mathrm dx = \frac{\pi}{2} \frac{r^{2m+1}}{2m+1} , \quad \mathrm |r| <1,$$ and $$J(m,r) = \int_{0}^{\pi/2} \cos\left((2m+1)x \right) \ln \left(\frac{1+2r \cos(x)+r^{2}}{1-2r \cos(x)+r^{2}} \right) \, \mathrm dx = \pi \, \frac{r^{2m+1}}{2m+1}, \quad |r| <1. $$
I'll prove the second formula.
The first formula can be derived in the same manner using the Fourier series for $\arctan \left(\frac{2r \sin x}{1-r^{2}} \right)$ obtained in Sangchul Lee's answer here.
For $|z| < 1$, $\ln(1-z)$ has the Taylor series $$\ln(1-z) = -\sum_{n=1}^{\infty} \frac{z^{n}}{n}. $$
Replacing $z$ with $re^{i x}$, where $|r|< 1$ and $x \in \mathbb{R}$, and then equating the real parts on both sides of the equation, we get $$\sum_{n=1}^{\infty} \frac{r^{n} \cos(nx)}{n} = -\frac{1}{2} \, \ln \left(1-2r \cos(x)+r^{2} \right). $$
Therefore, $$ \begin{align} \sum_{n=0}^{\infty} \frac{r^{2n+1} \cos\left((2n+1)x \right)}{2n+1} &= \frac{1}{2} \left( \sum_{n=1}^{\infty} \frac{r^{n} \cos(nx)}{n} - \sum_{n=1}^{\infty} \frac{(-1)^{n}r^{n} \cos(nx)}{n} \right) \\ &= \frac{1}{4} \, \ln \left(\frac{1+2r \cos(x)+r^{2}}{1-2r \cos(x)+r^{2}} \right), \end{align}$$
and $$ \begin{align} J(m,r) &= \int_{0}^{\pi/2} \cos\left((2m+1)x \right) \ln \left(\frac{1+2r \cos(x)+r^{2}}{1-2r \cos(x)+r^{2}} \right) \, \mathrm dx \\ &= 4 \int_{0}^{\pi/2} \cos \left((2m+1)x \right) \sum_{n=0}^{\infty} \frac{r^{2n+1} \cos\left((2n+1)x \right)}{2n+1} \, \mathrm dx\\ &= 4 \sum_{n=0}^{\infty} \frac{r^{2n+1}}{2n+1} \int_{0}^{\pi/2} \cos \left((2m+1)x \right) \cos \left((2n+1)x \right) \, \mathrm dx. \end{align}$$
If $n \ne m$, the value of the integral is zero. If $n=m$, the value of the integral is $\frac{\pi}{4}$.
This leads to the result.
