I wish to calculate $$I(R)=\int_0^{2\pi}\frac{\cos(\phi)-R}{1-2R\cos(\phi)+R^2}~\cos(n\phi)~d\phi,$$ where $n\in\mathbb{N}$, $R\in[0,1)$.
Based on trial and error from plugging numbers into Wolfram alpha I think the answer is $$I(R)=\begin{cases} 0, & n=0, \\ \pi R^{n-1}, & n\ge1. \end{cases}$$ However, I don't know how to show this more rigorously. Do you know how to calculate the above integral, and do you know if my formula is correct?
Given $R\in[0,1)$, there is a unique function with the following Fourier series:
$$ \sum_{n\geq 1}R^n \cos(n\theta) = \text{Re}\sum_{n\geq 1} R^n e^{in\theta} = \text{Re}\left(\frac{R e^{i\theta}}{1-R e^{i\theta}}\right)= \text{Re}\left(\frac{R e^{i\theta}(1-R e^{-i\theta})}{(1-R e^{i\theta})(1-Re^{-i\theta})}\right)$$ and such function is precisely $$ \frac{R\cos\theta-R^2}{1-2R\cos\theta+R^2}. $$
Define: $z=e^{i\phi}$
$$\cos(\phi)=\frac{1}2\left(z+\frac{1}z \right),~~~~\cos(n\phi)=\frac{1}2\left(z^n+\frac{1}{z^n} \right),~~~~d\phi=\frac{1}{iz}dz$$ If $n=0$, $$\begin{align} I&=\frac{1}{2i}\oint \frac{z^2-2Rz+1}{z(z-R)(1-Rz)}dz=\frac{1}{2i}\cdot 2\pi i\cdot \left(Res[z=0]+Res[z=R]\right)\\ \\ &=\pi\cdot\left( -\frac{1}R+\frac{1-R^2}{R(1-R^2)}\right)=0 \end{align}$$
If $n\ge 1$ $$\begin{align} I&=\frac{1}{4i}\oint \frac{(z^2-2Rz+1)z^{n-1}}{(z-R)(1-Rz)}dz+\frac{1}{4i}\oint \frac{(z^2-2Rz+1)}{z^{n+1}(z-R)(1-Rz)}dz=I_1+I_2\\ \\ I_1&=\frac{1}{4i}\cdot 2\pi i\cdot Res[z=R]=\frac{\pi}2R^{n-1}\\ \\ I_2&=\frac{1}{4i}\cdot 2\pi i\cdot \left(Res[z=0]+Res[z=R] \right)=\frac{\pi}2 \left( -\frac{1}{R^{n+1}}+R^{n-1}+\frac{1}{R^{n+1}} \right)\\ \\ I&=\pi\cdot R^{n-1} \end{align}$$
Let $z=e^{i\phi}$ and then $$ dz=izd\phi, \cos(n\phi)=\frac12\bigg(z^n+\frac{1}{z^n}\bigg). $$ Observing $$ \Re\frac{1}{z-R}=\Re\frac{\bar z-R}{|z-R|^2}=\frac{\cos\phi-R}{1-2R\cos\phi+R^2}$$ one has \begin{eqnarray} I(R)&=&\int_0^{2\pi}\frac{\cos(\phi)-R}{1-2R\cos(\phi)+R^2}~\cos(n\phi)~d\phi\\ &=&\frac12\Re \int_{|z|=1}\frac{1}{z-R}\bigg(z^n+\frac{1}{z^n}\bigg)\frac{1}{iz}dz\\ &=&-\frac12\Re i\int_{|z|=1}\bigg(\frac{z^{n-1}}{z-R}+\frac{1}{z^{n+1}(z-R)}\bigg)dz. \end{eqnarray} So if $n=0$ $$ I(R) = -\frac12\Re i\int_{|z|=1}\frac{2}{z(z-R)}dz=0.$$ If $n\ge 1$, $$ I(R) = -\frac12\Re i\int_{|z|=1}\bigg(\frac{z^{n-1}}{z-R}+\frac{1}{z^{n+1}(z-R)}\bigg)dz=-\frac12\Re i\int_{|z|=1}\frac{z^{n-1}}{z-R}dz=\pi R^{n-1}.$$
$$ \int_{0}^{2\pi} \frac{(\cos(x) - R)\cos(nx)}{1-2R\cos(x)+R^2 }\mathrm{d}x \overset{x\to x-\pi}{=}2(-1)^{n+1}\int_{0}^{\pi} \frac{\frac{\cos((n-1)x)}{2} + \frac{\cos((n+1)x)}{2} + R\cos(nx)}{1+2R\cos(x)+R^2 }\mathrm{d}x $$ And from this answer we know $$ \int_0^\pi\frac{\cos(mx)}{a^2-2ab\cos (x)+b^2}\ \mathrm{d}x=\frac{\pi}{a^2-b^2}\left(\frac{b}{a}\right)^m\quad\hbox{for}\quad |b|<a , \, m \in \mathbb{Z}\cap[0, \infty) $$ So for $|R| <1$ and $n \ge 1$ we have \begin{align} I(R) & = \frac{2 (-1)^{n+1}\pi}{1-R^2}\left[\frac{(-R)^{n-1}}{2} + \frac{(-R)^{n+1}}{2} + R(-R)^n\right] = \pi R^{n-1} \end{align} And for the $n=0$ case, since $I = -2 \int_0^{\pi} \frac{\cos(x)+R}{1+R^2 +2R\cos(x)}\mathrm{d}x$ then $$ I(0) = \frac{-2\pi}{1-R^2} \left[(-R)^1 +R(-R)^0 \right] =0 $$
For $n \in \mathbb{N}$ and $R \in \left[0,1\right)$, we can rewrite the integral using the complex definition of $\cos{(\phi)}$ and taking the real part as follows:
$$\eqalign{ \int_{0}^{2\pi}\frac{\left(\cos\left(\phi\right)-R\right)\cos\left(n\phi\right)}{1-2R\cos\left(\phi\right)+R^{2}}d\phi &= \Re\int_{0}^{2\pi}\frac{e^{in\phi}\left(\frac{e^{i\phi}+e^{-i\phi}}{2}-R\right)}{1-2R\left(\frac{e^{i\phi}+e^{-i\phi}}{2}\right)+R^{2}}d\phi. }$$
Letting $z = e^{i\phi}$, introducing the unit circle contour (let's call is $C$), traversing in the positive direction, and doing a lot of algebra, we get the integral to equal
$$\Re\left(\frac{i}{2}\oint\frac{z^{n-1}\left(z^{2}-2Rz+1\right)}{\left(z-R\right)\left(Rz-1\right)}dz\right).$$
Finding the singularities, we set the denominator equal to $0$ to get the set of zeroes $z_0 \in \left\{R,\frac{1}{R}\right\}$. Since $R \in \left[0,1\right)$, we know that $z = \frac{1}{R}$ is not inside $C$, so we can make $z_0 = R$ our pole to take the residue. Then the integral equals
$$\Re\left(\frac{i}{2}2\pi i \operatorname{Res}\left(\frac{z^{n-1}\left(z^{2}-2Rz+1\right)}{\left(z-R\right)\left(Rz-1\right)}, z = R\right)\right) = \Re\left(-\pi\lim_{z \to R}(z-R)\frac{z^{n-1}\left(z^{2}-2Rz+1\right)}{\left(z-R\right)\left(Rz-1\right)}\right).$$
Taking that limit and simplifying leads to the integral to equal $\pi R^{n-1}.$ You can do a similar method for the $n=0$ case as well.
