Instead of starting from your integral, I will start from the integral in the solution by @Mickep, i.e,
$$\tag{1}I=4\pi\int_0^{\pi/2}\frac{1}{\phi-\cos^2x}\,dx=\pi\underbrace{\int_0^{2\pi}\frac{1}{\phi-\cos^2x}\,dx}_{J} $$
on which it will be simpler to explain how to compute an integral using residue calculus than on the original integral. I will not enter into details. For this I refer you to the numerous lecture notes on residue calculus, for example here.
The main idea is to transform this integral into a circuit integral along a closed parametrized arc, which is here, in a natural way, the unit circle $\gamma$, traversed in the direct orientation.
Let $z=e^{ix}$, with $dz=ie^{ix}dx$.
Thus, $J$ becomes, using Euler formula $\cos(x)=\frac12(e^{ix}+e^{-ix})$:
$$J=\int_{\gamma}\dfrac{-i dz/z}{\Phi-\dfrac{(z+1/z)^2}{4}}$$
Expanding and reducing, we get:
$$\tag{2}J=-i \underbrace{\int_{\gamma}\dfrac{4z dz}{-z^4+2(2\Phi-1)z^2-1}}_{K}$$
The integrand has four poles (roots of the denominator), all of them real:
$$\begin{cases}z_1=\sqrt{2 \Phi -3} \ \ \ \text{and } \ \ \ z_2=-z_1=-\sqrt{2 \Phi -3}\\
z_3=\sqrt{1+2 \Phi} \ \ \ \text{and } \ \ \ z_4=-z_3=-\sqrt{1+2 \Phi}\end{cases}$$
Only $z_1$ and $z_2$ are inside contour $\gamma.$
The residue theorem says that a complex integral $\int_{\gamma}...$ whatever the sufficiently regular closed contour $\gamma$, is equal to the sum of residues at the poles situated inside the contour multiplied by $2i\pi$.
Now, what is the residue of a function of the form $\dfrac{f(z)}{g(z)}$ at a pole $z_0$? It is the number given by the following formula:
$$\dfrac{f(z_0)}{g'(z_0)}.$$
Although this is not the most general definition, this expression covers a large number of cases (see remark below).
Thus, the residue theorem gives:
$$K=2i\pi\left(\dfrac{4z_1 }{-4z_1^3+4(2\Phi-1)z_1}+\dfrac{4z_2}{-4z_2^3+4(2\Phi-1)z_2}\right)$$
$$K=2i\pi\left(\dfrac{1}{-z_1^2+(2\Phi-1)}+\dfrac{1}{-z_2^2+(2\Phi-1)}\right)$$
As $z_1^2=z_2^2=2\Phi-3$, the denominators have a common value, which is $-2$.
Thus $K=-2i\pi$. Plugging this value into (2) and then (1) gives the awaited result.
Remark: The definition of residues I have given is only valid for simple poles; if $z_0$ is for example a double root of the denominator, it is understandable that we are in trouble because $g'(z_0)=0$. There exist specific formulas for these cases.