I would like to know if there is a higher-level perspective to explain this result (or a simple way to prove it)? Thanks for help.
$$\int_0^{2\pi} \left(a\cos x+\sqrt{1-a^2\sin^2(x)}\right)^{2k} dt=2\pi\sum_{m=1}^k \binom{k}{m} \binom{k-1}{m-1}a^{2(k-m)}.$$
Here's my progress toward a (very long-winded and messy) solution. I think it's mostly complete; there's just one binomial sum at the end that I haven't figured out how to evaluate.
The first thing we'll do is manipulate the left-hand side to get rid of the square root, since it's probably the most annoying part of the expression. $$\displaystyle\sum_{j=0}^{2n-1}\left[a\cos\left(\theta+\dfrac{j\pi}{n}\right)+\sqrt{1-a^2\sin^2\left(\theta+\dfrac{j\pi}n\right)}\right]^{2k}$$ $$= \displaystyle\sum_{j=0}^{n-1}\left[\left[a\cos\left(\theta+\dfrac{j\pi}{n}\right)+\sqrt{1-a^2\sin^2\left(\theta+\dfrac{j\pi}n\right)}\right]^{2k} + \left[a\cos\left(\theta+\dfrac{(j+n)\pi}{n}\right)+\sqrt{1-a^2\sin^2\left(\theta+\dfrac{(j+n)\pi}n\right)}\right]^{2k}\right]$$ $$= \displaystyle\sum_{j=0}^{n-1}\left[\left[a\cos\left(\theta+\dfrac{j\pi}{n}\right)+\sqrt{1-a^2\sin^2\left(\theta+\dfrac{j\pi}n\right)}\right]^{2k} + \left[-a\cos\left(\theta+\dfrac{j\pi}{n}\right)+\sqrt{1-a^2\sin^2\left(\theta+\dfrac{j\pi}n\right)}\right]^{2k}\right]$$ $$= \displaystyle\sum_{j=0}^{n-1}\left[\left[\sum_{m=0}^{2k}\binom{2k}{m}a^m\cos^m\left(\theta+\dfrac{j\pi}n\right)\left(1-a^2\sin^2\left(\theta+\dfrac{j\pi}n\right)\right)^{k-m/2}\right] + \left[\sum_{m=0}^{2k}\binom{2k}{m}(-a)^m\cos^m\left(\theta+\dfrac{j\pi}n\right)\left(1-a^2\sin^2\left(\theta+\dfrac{j\pi}n\right)\right)^{k-m/2}\right]\right]$$ $$= \displaystyle\sum_{j=0}^{n-1}\sum_{m=0}^{2k}\binom{2k}{m}\left(1+(-1)^m\right)a^m\cos^m\left(\theta+\dfrac{j\pi}n\right)\left(1-a^2\sin^2\left(\theta+\dfrac{j\pi}n\right)\right)^{k-m/2}$$ $$= \displaystyle\sum_{j=0}^{n-1}\sum_{m=0}^k\binom{2k}{2m}2a^{2m}\cos^{2m}\left(\theta+\dfrac{j\pi}n\right)\left(1-a^2\sin^2\left(\theta+\dfrac{j\pi}n\right)\right)^{k-m}$$
(In the last step, we switched to just summing over the even values of $m$, since the odd values vanish due to the $1+(-1)^m$.)
So now we've managed to get rid of that pesky square root. Now, since this problem has "roots of unity" written all over it, the next thing we'll try is interchanging the sums and trying to get the $j$ sum to simplify to something trivial.
$$\displaystyle\sum_{j=0}^{n-1}\sum_{m=0}^k\binom{2k}{2m}2a^{2m}\cos^{2m}\left(\theta+\dfrac{j\pi}n\right)\left(1-a^2\sin^2\left(\theta+\dfrac{j\pi}n\right)\right)^{k-m}$$ $$= 2\displaystyle\sum_{m=0}^k\binom{2k}{2m}a^{2m}\sum_{j=0}^{n-1}\cos^{2m}\left(\theta+\dfrac{j\pi}n\right)\left(1-a^2\sin^2\left(\theta+\dfrac{j\pi}n\right)\right)^{k-m}$$ $$= 2\displaystyle\sum_{m=0}^k\binom{2k}{2m}a^{2m}\sum_{j=0}^{n-1}\cos^{2m}\left(\theta+\dfrac{j\pi}n\right)\sum_{\ell=0}^{k-m}\binom{k-m}{\ell}(-1)^{\ell}a^{2\ell}\sin^{2\ell}\left(\theta+\dfrac{j\pi}n\right)$$ $$= 2\displaystyle\sum_{m=0}^k\sum_{\ell=0}^{k-m}\binom{2k}{2m}\binom{k-m}{\ell}(-1)^{\ell}a^{2\ell+2m}\sum_{j=0}^{n-1}\cos^{2m}\left(\theta+\dfrac{j\pi}n\right)\sin^{2\ell}\left(\theta+\dfrac{j\pi}n\right)$$
And now we concentrate on just the innermost sum. In the following, let $z=e^{i\theta}$ and $\omega=e^{\frac{i\pi}n}$. (So $\omega$ is a $2n$-th root of unity.) We note by Euler's Identity that $$\cos\left(\theta+\dfrac{j\pi}n\right) = \dfrac{z\omega^j + \frac1{z\omega^j}}2$$ and $$\sin\left(\theta+\dfrac{j\pi}n\right) = \dfrac{z\omega^j - \frac1{z\omega^j}}{2i}$$ So the inner sum becomes $$\sum_{j=0}^{n-1}\cos^{2m}\left(\theta+\dfrac{j\pi}n\right)\sin^{2\ell}\left(\theta+\dfrac{j\pi}n\right)$$ $$= \sum_{j=0}^{n-1}\left(\dfrac{z\omega^j + \frac1{z\omega^j}}2\right)^{2m}\left(\dfrac{z\omega^j - \frac1{z\omega^j}}{2i}\right)^{2\ell}$$ $$= \dfrac{(-1)^{\ell}}{4^{\ell+m}}\sum_{j=0}^{n-1}\left(z\omega^j + \dfrac1{z\omega^j}\right)^{2m}\left(z\omega^j - \dfrac1{z\omega^j}\right)^{2\ell}$$ $$= \dfrac{(-1)^{\ell}}{4^{\ell+m}}\sum_{j=0}^{n-1}\left[\sum_{r=0}^{2m}\binom{2m}{r}\left(z\omega^j\right)^r\left(\dfrac1{z\omega^j}\right)^{2m-r}\right]\left[\sum_{s=0}^{2\ell}\binom{2\ell}{s}\left(z\omega^j\right)^s\left(-\dfrac1{z\omega^j}\right)^{2\ell-s}\right]$$ $$= \dfrac{(-1)^{\ell}}{4^{\ell+m}}\sum_{j=0}^{n-1}\sum_{r=0}^{2m}\sum_{s=0}^{2\ell}\binom{2m}{r}\binom{2\ell}{s}(-1)^s\left(z\omega^j\right)^{2(r+s-\ell-m)}$$ $$= \dfrac{(-1)^{\ell}}{4^{\ell+m}}\sum_{r=0}^{2m}\sum_{s=0}^{2\ell}\binom{2m}{r}\binom{2\ell}{s}(-1)^sz^{2(r+s-\ell-m)}\left[\sum_{j=0}^{n-1}\left(\omega^2\right)^{j(r+s-\ell-m)}\right]$$
Now, since $\omega^2$ is an $n$-th root of unity, the sum in brackets above is zero... unless $r+s-\ell-m$ is a multiple of $n$, in which case every term of that sum turns out to be 1, which gives a grand total of $n$ instead. Now, we need to look at our nested indices and figure out how large $r+s-\ell-m$ can get.
We know $0\le m\le k$, $0\le\ell\le k-m$, $0\le r\le 2m$, and $0\le s\le 2\ell$. The sum $r+s-\ell-m$ is minimized when $r=s=0$, and then since $\ell\le k-m$, $\ell+m\le k$, so the minimum is at $-k$. SImilarly, the sum is maximized when $r=2m$ and $s=2\ell$, which gives $\ell+m\le k$. Finally, $k<n$. So the only possible value of $r+s-\ell-m$ that is divisible by $n$ is $0$ itself, since $-n<r+s-\ell-m<n$.
Now we're going to rework the sum so that we only use values of $r$ and $s$ such that $r+s=\ell+m$. Then our sum is
$$= \dfrac{(-1)^{\ell}}{4^{\ell+m}}\sum_{r=0}^{2m}\sum_{s=0}^{2\ell}\binom{2m}{r}\binom{2\ell}{s}(-1)^sz^{2(r+s-\ell-m)}\left[\sum_{j=0}^{n-1}\left(\omega^2\right)^{j(r+s-\ell-m)}\right]$$ $$= \dfrac{(-1)^{\ell}}{4^{\ell+m}}\sum_{0\le r\le 2m;\,0\le s\le 2\ell;\,r+s=\ell+m}\binom{2m}{r}\binom{2\ell}{s}(-1)^sn$$
Now, we let $t=r-m=\ell-s$. Then $-m\le t\le m$, and $-\ell\le t\le\ell$. Using this information, we can simplify to a single sum: $$\dfrac{n}{4^{\ell+m}}\sum_{t=-\min(\ell,m)}^{\min(\ell,m)}\binom{2m}{m+t}\binom{2\ell}{\ell+t}(-1)^t$$
Since the sum is symmetric, we can assume WLOG that $m\le \ell$ to get rid of the annoying index constraint. $$\dfrac{n}{4^{\ell+m}}\sum_{t=-m}^{m}\binom{2m}{m+t}\binom{2\ell}{\ell+t}(-1)^t$$ $$=\dfrac{(-1)^mn}{4^{\ell+m}}\sum_{t=0}^{2m}\binom{2m}{t}\binom{2\ell}{\ell-m+t}(-1)^t$$ $$=\dfrac{(-1)^mn}{4^{\ell+m}}\dfrac{(2m)!(2\ell)!}{(m+\ell)!^2}\sum_{t=0}^{2m}\binom{\ell+m}{t}\binom{\ell+m}{2m-t}(-1)^t$$
And now we consider the binomial expansion of $(1+x)^{\ell+m}(1-x)^{\ell+m}$. The coefficient of $x^{2m}$ in this expansion is given by the sum $$\sum_{t=0}^{2m}\binom{\ell+m}{t}\binom{\ell+m}{2m-t}(-1)^t$$ due to expanding with the binomial theorem. Meanwhile, if we multiply using the difference of squares, this simplifies to $(1-x^2)^{\ell+m}$, which shows that $x^{2m}$ has a coefficient of $(-1)^m\binom{\ell+m}{m}$. So the two must be equal: $$\sum_{t=0}^{2m}\binom{\ell+m}{t}\binom{\ell+m}{2m-t}(-1)^t=(-1)^m\binom{\ell+m}{m}$$. (The above identity is stated as Equation 1.19 in [url=http://www.math.wvu.edu/~gould/Vol.4.PDF]this PDF[/url], which is where I eventually had to resort to looking in trying to evaluate this stubborn sum.)
Now that we have a closed form for our sum, we plug it back in. $$\dfrac{n}{4^{\ell+m}}\sum_{t=-\min(\ell,m)}^{\min(\ell,m)}\binom{2m}{m+t}\binom{2\ell}{\ell+t}(-1)^t = \dfrac{n}{4^{\ell+m}}\dfrac{(2m)!(2\ell)!}{(m+\ell)!^2}\binom{\ell+m}{\ell} = \dfrac{n}{4^{\ell+m}}\dfrac{(2m)!(2\ell)!}{m!\ell!(m+\ell)!}$$
But this is all just the evaluation of the innermost sum with $j$. The original problem still has two more layers of summation! With our simplification, the problem now becomes $$2n\displaystyle\sum_{m=0}^k\sum_{\ell=0}^{k-m}\binom{2k}{2m}\binom{k-m}{\ell}\dfrac{(-1)^{\ell}a^{2(\ell+m)}}{4^{\ell+m}}\dfrac{(2m)!(2\ell)!}{m!\ell!(m+\ell)!}$$ This is finally starting to look kind of like the expression on the right! Now, a change of variables is in order. We let $p=\ell+m$. (At this point, we're really starting to run out of letters of the alphabet to use for variable names...) Then, after suitably changing the indices in the double sum, we obtain $$2n\displaystyle\sum_{p=0}^k\sum_{\ell=0}^{p}\binom{2k}{2p-2\ell}\binom{k+\ell-p}{\ell}\dfrac{(-1)^{\ell}a^{2p}}{4^p}\dfrac{(2p-2\ell)!(2\ell)!}{p!\ell!(p-\ell)!}$$ $$=2n\displaystyle\sum_{p=0}^k\dfrac{a^{2p}}{4^pp!}\left[\sum_{\ell=0}^{p}\binom{2k}{2p-2\ell}\binom{k+\ell-p}{\ell}\dfrac{(-1)^{\ell}(2p-2\ell)!(2\ell)!}{\ell!(p-\ell)!}\right]$$
We focus now on the inner sum. $$\displaystyle\sum_{\ell=0}^{p}\binom{2k}{2p-2\ell}\binom{k+\ell-p}{\ell}\dfrac{(-1)^{\ell}(2p-2\ell)!(2\ell)!}{\ell!(p-\ell)!}$$ $$=\dfrac{(2k)!}{(k-p)!}\displaystyle\sum_{\ell=0}^{p}\binom{2\ell}{\ell}(-1)^{\ell}\dfrac{(k+\ell-p)!}{(2k+2\ell-2p)!(p-\ell)!}$$ $$=\dfrac{(2k)!}{k!(k-p)!}\displaystyle\sum_{\ell=0}^{p}(-1)^{\ell}\dfrac{\binom{2\ell}{\ell}\binom{k}{p-\ell}}{\binom{2k+2\ell-2p}{k+\ell-p}}$$
It remains to establish the following identity: $$\sum_{\ell=0}^{p}(-1)^{\ell}\dfrac{\binom{2\ell}{\ell}\binom{k}{p-\ell}}{\binom{2k+2\ell-2p}{k+\ell-p}} = \dfrac{4^p\binom{k-1}{p}}{\binom{2k}{k}}$$ which can be easily checked for small values of $p$. (We use the convention that, if $p>k-1$, then $\binom{k-1}{p}=0$.)
[b]If the above identity, which I have not yet been able to prove rigorously, is true[/b], then the outer sum becomes $$2n\displaystyle\sum_{p=0}^k\dfrac{a^{2p}}{4^pp!}\left[\dfrac{(2k)!}{k!(k-p)!}\dfrac{4^p\binom{k-1}{p}}{\binom{2k}{k}}\right]$$ $$=2n\displaystyle\sum_{p=0}^ka^{2p}\binom{k}{p}\binom{k-1}{p}$$ which becomes the desired expression after substituting $p=k-m$ and disregarding the $m=0$ case.
