$$\int_0^\pi {\sin x\sin nx\over 1-2a\cos x+a^2} dx$$
With some bit of tinkering with Desmos, I've got to know that the answer is ${\pi \over 2} a^{n-1}$.
But can you help prove that?
Sorry for the typo. I meant $\sin nx$ instead of $\cos nx$.
$$\int_0^\pi {\sin x\sin nx\over 1-2a\cos x+a^2} dx$$
With some bit of tinkering with Desmos, I've got to know that the answer is ${\pi \over 2} a^{n-1}$.
But can you help prove that?
Sorry for the typo. I meant $\sin nx$ instead of $\cos nx$.
We can do something more general using a result proven here, namely: $$\int_0^\pi\frac{\cos(mx)}{a^2-2ab\cos x+b^2}dx=\frac{\pi}{a^2-b^2}\left(\frac{b}{a}\right)^m$$
$$\int_0^\pi \frac{\sin(kx)\sin(n x)}{a^2-2ab\cos x+b^2}dx=\frac12\int_0^\pi \frac{\cos((k-n)x)-\cos((k+n)x)}{a^2-2ab\cos x+b^2}dx$$ $$=\frac{\pi}{2(a^2-b^2)}\left(\left(\frac{b}{a}\right)^{n-k}-\left(\frac{b}{a}\right)^{n+k}\right),\quad n>k;\ \ a>b>0.$$ Just swap $(n,k)$ and $(a,b)$ for other conditions.
Partial Answer
This function is even so we can write$$\int_0^\pi {\sin x\sin nx\over 1-2a\cos x+a^2} dx={1\over 2}\int_{-\pi}^\pi {\sin x\sin nx\over 1-2a\cos x+a^2} dx$$by defining $z=e^{ix}$ we have$$\int_{-\pi}^\pi {\sin x\sin nx\over 1-2a\cos x+a^2} dx{=\oint_{|z|=1} {{1\over 2i}(z-z^{-1}){1\over 2i}(z^{n}-z^{-n})\over 1-a\left(z+{1\over z}\right)+a^2} {dz\over iz}\\=-{1\over 4}\oint_{|z|=1} {(z-z^{-1})(z^{n}-z^{-n})\over 1-a\left(z+{1\over z}\right)+a^2} {dz\over iz}\\=-{1\over 4}\oint_{|z|=1} {(z^2-1)(z^{2n}-1)\over z^n((1+a^2)z-a\left(z^2+1\right))} {dz\over iz}\\={1\over 4ai}\oint_{|z|=1} {(z^2-1)(z^{2n}-1)\over z^{n+1}(z-a)\left(z-{1\over a}\right)} {dz}}$$with the singularities of ${(z^2-1)(z^{2n}-1)\over z^{n+1}(z-a)\left(z-{1\over a}\right)}$ falling in $z={1\over a},0,a$ when $a\ne 1$. For $r=1$, the only singularity exists in $z=0$.
$$I=\int_{0}^{\pi} \frac {\sin x \sin n x}{1-2a \cos x+a^2} dx ~~~~(1)$$ Integrate it by parts taking $\sin nx$ as the first function, we get $$I=-\frac{n}{2a}\int_{0}^{\pi} \cos n x \ln[1-2a \cos x +a^2] dx ~~~~(2).$$ Let $y=\cos x+i \sin x$, then $$f(x)=\ln(1-2a\cos x+a^2)=ln(1-ay)+\ln(1-a/y).$$ If $a^2<1,$ then $$f(x)=-2[a \cos x + \frac{a^2\cos 2 x}{2}+\frac{a^3 \cos 3 x}{3}+...]~~~~(3)$$
Using (3) in (2) and using the property that $$\int_{0}^{\pi} \cos m x \cos n x dx=\frac{\pi}{2}\delta_{m,n}$$ we get $$I=\frac{n}{a} \frac{a^n}{n} \frac{\pi}{2}=\frac{\pi a^{n-1}}{2},~~~ a^2 <1.$$