Suppose WLOG $b > a$. Then with $c = b/a$ we have
$$\begin{align} I &= \int_0^\pi \log(a^2 + b^2 - 2ab \cos x) \, dx \\ &=\int_0^\pi \log a^2 \, dx + \int_0^\pi \log(1 + (b/a)^2 - 2(b/a) \cos x) \, dx \\ &= 2\pi \log a + \int_0^\pi \log(1 + c^2 - 2c \cos x) \, dx \end{align}$$
Now we can evaluate the second integral on the RHS as the limit of a Riemann sum
$$\begin{align}J &= \int_0^\pi \log(1 + c^2 - 2c \cos x) \, dx \\ &= \lim_{n \to \infty} \frac{\pi}{n}\sum_{j=1}^n\log(1 + c^2 - 2c\cos(\pi(j-1)/n)) \\ &= \lim_{n \to \infty} \frac{\pi}{n}\log\prod_{j=1}^n(1 + c^2 - 2c\cos(\pi(j-1)/n)) \\ &=\lim_{n \to \infty} \frac{\pi}{n}\log(1-c)^2+\lim_{n \to \infty} \frac{\pi}{n}\log\prod_{j=2}^{n}(1 + c^2 - 2c\cos(\pi(j-1)/n)) \\ &= \lim_{n \to \infty} \frac{\pi}{n}\log\prod_{j=1}^{n-1}(1 + c^2 - 2c\cos(\pi j/n))\end{align} .$$
Upon factoring, we obtain
$$1 + c^2 - 2c\cos(\pi j/n) = [c - \exp(i \pi j /n)][c - \exp(-i \pi j /n)],$$
and
$$c^{2n} -1 = (c-1)(c+1)\prod_{j=1}^{n-1}[c - \exp(i \pi j /n)][c - \exp(-i \pi j /n)].$$
Hence, recalling that $c = b/a > 1$, it follows that
$$\begin{align}J &= \lim_{n \to \infty} \frac{\pi}{n} \log\left(\frac{c^{2n}-1}{c^2-1} \right) \\ &= \pi \lim_{n \to \infty} \log\left(\frac{c^{2n}-1}{c^2-1} \right)^{1/n} \\ &= \pi \lim_{n \to \infty} \log\left[c^2 \left(\frac{1-c^{-2n}}{c^2-1} \right)^{1/n}\right] \\ &= \pi \log c^2 + \log \left[\lim_{n \to \infty} \left(\frac{1-c^{-2n}}{c^2-1} \right)^{1/n}\right]\\ &= \pi \log c^2\end{align}$$
Thus,
$$I = 2\pi \log a + \pi \log c^2 = 2\pi \log a + \pi \log (b/a)^2 \\ = 2\pi \log b $$