I think there was a typo in that paper you read.
Entry 4.324.2 in Gradshteyn and Ryzhik says that the value of $I$ is $ \log(1+\alpha) \log \left(\frac{q^{2}}{p^{2}} \right)$ when $-1 <\alpha \le 1$, and $\log \left(1+ \frac{1}{\alpha} \right)\log \left(\frac{q^{2}}{p^{2}} \right) $ when $\alpha^{2} >1$.
I will show the first case.
Let $$I(s) = \int_{0}^{\infty}x^{s-1} \log \left(\frac{1+2\alpha \cos(px)+\alpha^{2}}{1+2 \alpha \cos(qx) + \alpha^{2}} \right) \, \mathrm dx,$$ where $-2 < s < 0$, $\alpha^{2} <1$, and $p$ and $q$ are positive real numbers.
The integral also converges for $0 \le s < 1$, but those values are purposely not included so that we can use Fubini's theorem to justify interchanging the order of summation and integration below.
It then follows from a property of the Mellin transform that $$I = \int_{0}^{\infty} \frac{1}{x} \, \log \left(\frac{1+2\alpha \cos(px)+\alpha^{2}}{1+2 \alpha \cos(qx) + \alpha^{2}} \right) \, \mathrm dx = \lim_{s \to 0^{-}} I(s).$$
Using the Fourier series $$\log(1+2\alpha \cos (\theta) + \alpha^{2}) = -2 \sum_{k=1} \frac{(-\alpha)^{k}\cos(k \theta)}{k}, \quad \alpha^{2}<1,$$
and the Mellin transform $$\int_{0}^{\infty}x^{s-1} \left( \cos(px) - \cos(qx) \right)\, \mathrm dx = \Gamma(s) \cos \left(\frac{\pi s}{2} \right) \left(p^{-s}-q^{-s} \right), \quad -2 <s <1, $$ we have
$$ \begin{align} I(s) &= -2 \int_{0}^{\infty} x^{s-1}\sum_{k=1}^{\infty} \left( \frac{(-\alpha)^{k} \cos(kpx)}{k} - \frac{(-\alpha)^{k}\cos(kqx)}{k} \right)\, \mathrm dx \\ &= -2 \sum_{k=1}^{\infty} \frac{(-\alpha)^{k}}{k} \int_{0}^{\infty} x^{s-1} \left(\cos(kpx)-\cos(kqx) \right) \, \mathrm dx \\ &= - 2 \sum_{k=1}^{\infty} \frac{(-\alpha)^k}{k} \Gamma(s) \cos \left(\frac{\pi s}{2} \right)\left((kp)^{-s}-(kq)^{-s} \right) \\ &=-2 \, \Gamma(s) \cos \left(\frac{\pi s}{2} \right) \left(p^{-s}-q^{-s} \right) \sum_{k=1}^{\infty} \frac{(-\alpha)^{k}}{k^{s+1}} \\ &= 2 \, \Gamma(s) \cos \left(\frac{\pi s}{2} \right) \left(q^{-s}-p^{-s} \right) \operatorname{Li}_{s+1}(-\alpha). \end{align}$$
Therefore, $$ \begin{align} I &= \lim_{s \to 0^{-}} I(s) \\ &= \lim_{s \to 0^{-}} 2 \left(\frac{1}{s} + \mathcal{O}(s) \right)\cos \left(\frac{\pi s}{2} \right) \left(s (\log p -\log q) + \mathcal{O}(s^{2}) \right)\operatorname{Li}_{s+1}(-\alpha) \\ &= 2( \log p - \log q) \operatorname{Li}_{1}(-\alpha) \\ &= -2 ( \log p - \log q) \log(1+\alpha) \\ &= \log(1+\alpha) \log \left(\frac{q^{2}}{p^{2}} \right). \end{align}$$
UPDATE:
If $\alpha^{2} <1$, then $$\int_{0}^{n \pi/p} \log \left(1+2 \alpha \cos(px)+ \alpha^{2}\right) =0, \quad n \in \mathbb{N}_{\ge 0},$$ which means that $I$ converges by Dirichlet's test.
The case $\alpha^{2}>1$ comes from replacing $\alpha$ with $\frac{1}{\alpha}$ in the Fourier series.