1

It's my understanding that the Kullback-Leibler divergence can never be negative. Denoting the divergence as: $$D(P||Q) = E_{p}[\log(p(X) / q(X))] = \int p(x) \log(p(x) / q(x)) dx$$ We can then use Jensen's inequality as follows: $$E_{p}[\log(p(X) / q(X))] = -E_{p}[\log(q(X) / p(X))] \ge \log(E_{p}[q(X) / p(X)]) = \log\left(\int q(x)dx\right) = \log(1) = 0$$ However, I seem to be getting a counterexample in the Kullback-Leibler divergence between a $\text{Laplace}(\mu, b_{1})$ (the $P$ distribution) and $\text{Laplace}(\mu, b_{2})$ (the $Q$ distribution). Knowing that if $X$ has a $\text{Laplace}(\mu, b)$ distribution, $E[|X - \mu|] = b$, my argument was as follows: $$D(P||Q) = E_{p}[\log(p(X) / q(X))] = E_{p}\left[\left(\log\left(\frac{1}{2b_1}\right) - \frac{1}{b_1}|X - \mu|\right) - \left(\log\left(\frac{1}{2b_2}\right) - \frac{1}{b_2}|X - \mu|\right)\right] = \log\left(\frac{b_2}{b_1}\right) - 1 + \frac{b_1}{b_2}$$ However, this Desmos graph seems to imply that negative KL divergences are possible for some values of $(b_1, b_2)$. Suspecting I made a mistake somewhere, I looked for someone else's derivation. Stumbled across this, where they computed the KL divergence between a $\text{Laplace}(\mu_1, b_1)$ and $\text{Laplace}(\mu_2, b_2)$. My above formula is just a special case of theirs, with $\mu_1 = \mu_2$.

What am I missing here?

Giorgos Giapitzakis
  • 6,416
user62348
  • 33

1 Answers1

1

The formula you got is correct for $\log$ in base $e$ (natural log). Sadly, Desmos understands that $\log = \log_{10}$. Replace that by $\ln$ and you'll get the right graph.

Besides (using natural log): it's well known that

$$\log x \ge 1-\frac{1}{x} \implies \log x -1 + \frac{1}{x} \ge 0$$

for any $x >0$. Setting $x=b_2/b_1$ you get the desired property.

leonbloy
  • 63,430