I do not know what you've done, but if I had to guess, perhaps you didn't take into account the double poles.
Rewrite the integral as
$$I = \frac12 \int_0^{2 \pi} \frac{d\theta}{(a+b \cos{\theta})^2}$$
Now make the sub $z=e^{i \theta}$, $d\theta = dz/(i z)$:
$$I = -i 4 \frac12 \oint_{|z|=1} \frac{dz \, z}{(b z^2 + 2 a z+b)^2}$$
The poles of the integrand are at
$$z_{\pm} = -\frac{a}{b} \pm \sqrt{\frac{a^2}{b^2}-1}$$
The only pole inside the circle $|z|=1$ is $z=z_+$.Because there is a double pole at $z=z_+$, the residue at that pole is
$$-i \frac{2}{b^2} \left[\frac{d}{dz} \frac{z}{(z-z_-)^2} \right]_{z=z_+} = -i \frac{2}{b^2} \left [\frac{1}{(z_+-z_-)^2} - \frac{2 z_+}{(z_+-z_-)^3} \right ] $$
After some algebra which I leave to the reader, I get, for the residue
$$-i \frac{2}{b^2} \frac{a}{4 b} \left [ \frac{a^2}{b^2}-1\right]^{-3/2} $$
The integral is therefore $i 2 \pi$ times this residue, or
$$I = \pi a \left ( a^2-b^2\right)^{-3/2} $$