The question is to evaluate $$\int \log(1+2m \cos x+m^2) dx$$ when $m > 0$
I noticed that $(1+2m \cos x+m^2)$ can be written as $(1+me^{-x})(1+me^{ix})$ and then I can apply the Taylor expansion of $\log(x)$ but I am not sure if the approach is correct.
I'm not so sure of indefinite integration, but if we add bounds, it comes out beautifully, as shown here.
Take the derivative with respect to $m$ to get
$$\begin{align}-I'(-m)&=\int_0^\pi\frac{2m-2\cos(x)}{1-2m\cos(x)+m^2}\ dx\\&=\frac1m\int_0^\pi1-\frac{1-m^2}{1-2m\cos(x)+m^2}\ dx\\&=\frac\pi m-\frac2m\tan^{-1}\left(\frac{1+m}{1-m}\tan(x/2)\right)\bigg|_{x=0}^{x=\pi}\\&=\begin{cases}0&|m|<1\\\frac{2\pi}m&|m|>1\end{cases}\end{align}$$
It thus becomes clear that
$$I(-m)=\begin{cases}C_1&|m|<1\\2\pi\ln|m|+C_2&|m|>1\end{cases}$$
By substituting in a few values, one may deduce that $C_1=C_2=0$, thus
$$\int_0^\pi\ln(1+2m\cos(x)+m^2)\ dx=\begin{cases}0&|m|<1\\2\pi\ln|m|&|m|>1\end{cases}$$
Mathematica Output
Mathematica instantly produces the following: $$\frac{1}{2} x \left(2 \log \left(m^2+2 m \cos (x)+1\right)-2 \log \left(\frac{m+e^{i x}}{m}\right)-2 \log \left(1+m e^{i x}\right)+i x\right)+i \operatorname{Li}_2\left(\frac{-e^{i x}}{m}\right)+i \operatorname{Li}_2\left(-e^{i x} m\right)+C$$ I am working on a way to prove this by expanding and greatly improving on Durgesh's answer, but I am not there yet. Should all else fail, converting each of the terms to integral form and combining should work. There are definitely patterns emerging in the output; for example, the derivatives of the two Dilogarithm terms in the output are the latter two logarithm terms present in the output, i.e. $$\frac{d}{dx} i \operatorname{Li}_2\left(\frac{-e^{i x}}{m}\right) = \log \left(\frac{m+e^{i x}}{m}\right)$$ $$\frac{d}{dx} i \operatorname{Li}_2\left(-e^{i x} m\right) = \log \left(1+m e^{i x}\right)$$
This is a well-known problem about the Poisson kernel.
Since $\log\|z\|=\text{Re}\log(z)$ and for every $n\in\mathbb{Z}$ we have $\int_{0}^{2\pi}e^{ni\theta}\,d\theta=2\pi\,\delta(n)$,
$$\forall r\in\mathbb{R},\quad \int_{0}^{2\pi}\log\|1-r e^{i\theta}\|\,d\theta = 2\pi \log\max(1,|r|),\tag{1}$$ $$\forall r\in\mathbb{R},\quad \int_{0}^{2\pi}\log(1+r^2-2r\cos\theta)\,d\theta = 4\pi\log\max(1,|r|).\tag{2}$$
The indefinite integral is related with the dilogarithm function, but I doubt you really need it.
$$\int \ln{\left(1+2m\cos{x}+2m^2\right)}dx=x\ln{(1+2m^2)}+\int\ln{\left(1+\frac{2m}{1+2m^2}\cos{x}\right)}dx=$$ where $\displaystyle \left|\frac{2m\cos{x}}{1+2m^2}\right|<1$
$\displaystyle =x\ln{(1+2m^2)}+\sum\limits_{n=1}^{\infty}\frac{(-1)^{n+1}}{n}\left(\frac{2m}{1+2m^2}\right)^n\int\cos^n{x}dx$
$\displaystyle =x\ln{(1+2m^2)}+\sum\limits_{n=1}^{\infty}\frac{(-1)^{n+1}}{2^nn}\left(\frac{2m}{1+2m^2}\right)^n\sum\limits_{k=0}^n\binom{n}{k}\frac{e^{i(n-2k)x}}{i(n-2k)}+C$
