3

Ok, so I know that

$\ln(1+e^x)\approx x$ when $x$ is large.

But what about

$\ln(1+e^x+e^y)$ when both $x$ and $y$ are large?

I can figure out cases when $x\gg y$ or $y\gg x$ since that simplifies to the earlier case. But, not the case when both are large. Are there any approximations in this example?

Greg
  • 682
  • When $x=y$ it's approximately $x+\ln 2$. – vadim123 Mar 06 '15 at 05:29
  • 2
    That's a nearly-trivial estimate for $\ln(1+e^x)$, incidentally - a much better one can be gotten just by saying $\ln(1+e^x)$ $= \ln(e^x(1+e^{-x}))$ $=\ln e^x+\ln(1+e^{-x})$ $\approx x+e^{-x}$. You can do something similar here, though the accuracy won't be as good. – Steven Stadnicki Mar 06 '15 at 05:49
  • A rough lower bound is $log(e^x+e^y)>log(2\sqrt(e^xe^y))=log(2)+\frac12 x+\frac12 y$ – Mark Viola Mar 06 '15 at 05:52
  • Got lower and upper bounds now. See the answer attached herein – Mark Viola Mar 06 '15 at 06:10
  • What is the purpose of your approximation? That would certainly determine the fitness of a specific answer. – Michael Grant Mar 08 '15 at 06:09
  • Greg, I divided the answer into a three part discussion and hope it is helpful for you. Please let me know if this provides insights into the issue of having separate approaches for different regions of interest. Part 3 gives global bounds that work for all $(x,y)$, while Parts 1 and 2 isolate subregions of interest ($e^{-(x-y)}$ is small in Part 1 whereas $-(x-y)$ is small in Part 2. – Mark Viola Mar 08 '15 at 07:55

3 Answers3

5

One can certainly write for large $x, y$ that $$\ln(1 + e^x + e^y) \approx \ln(e^x + e^y) = \ln(e^x) \ln(1 + e^{y - x}) = x + \ln(1 + e^{y - x}),$$ which quickly leads to the given formulas when $x$ and $y$ are far apart.

In the limit where they are close, that is when $\epsilon := y - x$ is close to zero, we have that $$\ln(1 + e^{\epsilon}) = \ln \left[1 + \left(1 + \epsilon + \frac{1}{2} \epsilon^2 + O(\epsilon^3)\right)\right] = \ln 2 + \frac{1}{2} \epsilon + \frac{1}{8} \epsilon^2 + O(\epsilon^3),$$ and so for $x, y \gg 0$ we have $$\ln(1 + e^x + e^y) \approx \ln2 + \frac{1}{2} (x + y) + \frac{1}{8} (x - y)^2 + O((x - y)^3).$$ (It turns out that the part of the approximation that is polynomial in $(x - y)$ is even, and in particular we can replace $O((x - y)^3)$ with $O((x - y)^4)$.) In the limit $y \to x$ (i.e., $\epsilon \to 0$) this is just $\log 2 + x$, recovering vadim123's observation in his comment.

Travis Willse
  • 99,363
  • Interesting, Thanks! – Greg Mar 06 '15 at 15:07
  • How does one proceed in the "intermediate" case? That is, if $x>>1$ and $y>>1$, $x$ is not "much bigger than $y$ and vice versa, but $x-y<1$? – Mark Viola Mar 06 '15 at 21:13
  • The radius of convergence of the series given for $\ln(1 + e^{\epsilon})$ and based at zero is $\pi$, so as long as $|x - y| < \pi$, we can produce an approximation of the form $\ln 2 + \frac{1}{2}(x + y) + P(x - y)$ where $P$ is some (it turns out, even) polynomial, though for $|x - y| \approx \pi$ we will need a high degree polynomial for reasonably accuracy. Over the range where $x > y$ but $x \not,!\gg y$, the function's qualitative behavior varies quite a lot, so in that sense there is probably no "simple" approximation that treats this whole range simultaneously. – Travis Willse Mar 07 '15 at 01:16
  • Travis, I divided the answer into a three part discussion and hope it is helpful for you. Please let me know if this provides insights into the issue of having separate approaches for different regions of interest. Part 3 gives global bounds that work for all $(x,y)$, while Parts 1 and 2 isolate subregions of interest ($e^{-(x-y)}$ is small in Part 1 whereas $-(x-y)$ is small in Part 2. – Mark Viola Mar 08 '15 at 07:56
  • @Dr.MV At a glance this looks quite reasonable to me. Do you have some sense of how good the bounds are, i.e., bounds on the error depending on $x$ and $y$? – Travis Willse Mar 08 '15 at 08:56
  • Travis, thanks for having a look. The upper bound is actually fairly robust as an approximation. The lower bound is rather crude until $x$ and $y$ are close (as expected). The first few terms in the series of exponentials provides a really solid (pseudo-global) approximation. Quite obviously, the approximation for "small" $x-y$ is limited to "small" $x-y$. – Mark Viola Mar 08 '15 at 15:18
1

We will break this into 3 parts.

Part $1$ provides derivation of an series expansion for $\log(1+e^x+e^y)$ in terms of powers of $e^{-\left(x-y\right)}$, where it is assumed that $e^x>1+e^y$. This is equivalent to assuming that $x-y>\log\left(1+e^{-y}\right)$

In Part $2$, we develop a series converges rapidly when $x-y$ is adequately small. This development is motivated since the series in Part 1 converges slowly when $x-y$ is small.

Part $3$ provides upper and lower bounds that are global (i.e., We impose no conditions on $x$ or $y$.).


PART $1$:

Without loss of generality, assume $x>y$. Let $M=e^x$ and $m=e^y$ and further assume that $m+1<M$ to write

$$\begin{align} 1+e^x+e^y&=1+m+M\\\\ &=M\left(1+\frac{m+1}{M}\right) \end{align}$$

Then$$\log(1+e^x+e^y)=\log(M)+\log\left(1+\frac{m+1}{M}\right)$$

Using the Taylor series for $\log(1+x)$ yields

$$\log(1+e^x+e^y)=\log(M)+\sum_{n=1}^{\infty} (-1)^{n+1}\frac{1}{n}\left(\frac{m+1}{M}\right)^n$$

Next, using the binomial theorem to expand $\left(\frac{m+1}{M}\right)^n$ yields

$$\begin{align} \log(1+e^x+e^y)&=\log(M)+\sum_{n=1}^{\infty} \sum_{p=0}^{n}(-1)^{n+1}{n \choose p} \frac{1}{n}\left(\frac{1}{M}\right)^nm^{p}\\\\ &=x+\sum_{n=1}^{\infty} \sum_{p=0}^{n}(-1)^{n+1}{n \choose p} \frac{1}{n}e^{-\left(nx-py\right)} \end{align}$$

Examining the first few terms reveals

$$\begin{align} \log(1+e^x+e^y)&=x+e^{-(x-y)}-\frac12e^{-2(x-y)}+\frac13e^{-3(x-y)}+e^{-x}\left(1-e^{-(x-y)}+e^{-2(x-y)}\right)\\\\ &+O\left(e^{-2x}\right) \end{align}$$

This provides a good series expansion in powers of $e^{-(x-y)}$ when $e^{-(x-y)}<1-\frac{1}{1+e^y}<1$. However, the speed of series convergence decreases with decreasing $x-y>0$.

Part $2$ considers the case in which $x-y$ is "small."


PART $2$:

To address the case for which $0<x-y<1$, let's reexamine the term $\log\left(1+\frac{m+1}{M}\right)$. Set $s=-(x-y)$, and $w=M^{-1}$, and consider the function

$$f(s)=\log\left(1+w+e^s\right)$$

We proceed to expand $f$ in a Taylor series. Evaluating the derivatives up to third order at $s=0$ yields

$$\begin{align} f^{\left(0\right)}(s)&=\log(2+w)\\\\ f^{\left(1\right)}(s)&=\frac{1}{2+w}\\\\ f^{\left(2\right)}(s)&=\frac{1}{2+w}-\frac{1}{\left(2+w\right)^2}\\\\ f^{\left(3\right)}(s)&=\frac{1}{2+w}-\frac{3}{\left(2+w\right)^2}+\frac{2}{\left(2+w\right)^3} \end{align}$$

Next, note that the terms $(2+w)^{-k}=\frac{1}{2^k}\left(1+\frac{w}{2}\right)^k$ can be expanded as $$(2+w)^{-k}=\frac{1}{2^k}\left(1+O\left(w\right)\right)$$ where only the leading term was retained under the tacit assumption that $x>>1$ so that $w=e^{-x}<<1$. This permits simplifying the derivatives of $f$ as

$$\begin{align} f^{\left(0\right)}(0)&=\log(2)\\\\ f^{\left(1\right)}(0)&=\frac12\\\\ f^{\left(2\right)}(0)&=\frac14\\\\ f^{\left(3\right)}(0)&=0 \end{align}$$

Finally, we can express $\log(1+e^x+e^y)$ as

$$\begin{align} \log(1+e^x+e^y)&=x + f(0) + f^{(1)}(0) \left(-(x-y)\right) + \frac12 f^{(2)}(0) \left(-(x-y)\right) + O\left(\left(x-y\right)^4\right) + O\left(e^{-x}\right)\\\\ &=\log(2)+\frac12(x-y)+\frac18(x-y)^2+O\left(\left(x-y\right)^4\right)+O\left(e^{-x}\right) \end{align}$$

since $f^{\left(3\right)}(0)=0$. This validates the answer posted by @Travis for small $x-y$ and adds the last term to provide a strict equality.


PART $3$: We may also obtain crude bounds as follows. For the upper bound we have

$$\begin{align} 1+e^x+e^y&=(e^x+e^y)\left(1+\frac{1}{e^x+e^y}\right)\\\\ &<2\max(e^x,e^y)\left(1+\frac{1}{2\min(e^x,e^y)}\right)\\\\ \end{align}$$

Now use (i) the series for $\log(1+x)$ and (ii) the fact that the $\log$ is a monotonically increasing function, which implies $\log\left(\max(a,b)\right)=\max\left(\log(a),\log(b)\right)$ to reveal

$$\log\left(1+e^x+e^y\right)<\log(2)+\max(x,y)+\frac12\max\left(e^{-x},e^{-y}\right)$$

Under the assumption that $x>y$, this upper bound becomes

$$\log\left(1+e^x+e^y\right)\le\log(2)+x+\frac12e^{-y}$$

For the lower bound, we have

$$\begin{align} 1+e^x+e^y&=(e^x+e^y)\left(1+\frac{1}{e^x+e^y}\right)\\\\ &\ge2\sqrt(e^xe^y)\left(1+\frac{1}{e^x+e^y}\right) \end{align}$$

Now use (i) the series for $\log(1+x)$ and (ii) the inequality for the geometric and arithmetic means to reveal

$$\begin{align} \log(1+e^x+e^y)&\ge \log(2)+\frac12\left(x+y\right)+\frac{1}{e^x+e^y}-\frac12\left(\frac{1}{e^x+e^y}\right)^2\\\\ &\ge\log\left(2\right)+\frac12\left(x+y\right)+\frac12\min\left(e^{-x},e^{-y}\right)-\frac18\ \max\left(e^{-2x},e^{-2y}\right) \end{align}$$

Under the assumption that $x>y$, this lower bound becomes

$$\log(1+e^x+e^y)\ge\log\left(2\right)+\frac12\left(x+y\right)+\frac12e^{-x}-\frac18\ e^{-2y}$$

Mark Viola
  • 179,405
0

As @TravisWillse noted in his answer, for positive $\{x,\ y\}\gg 1$ you will have: $$\begin{array}{r c l} \ln(1+e^x+e^y)&\approx &\ln(e^x+e^y)\\ & = & x+\ln(1+e^{y-x}) \end{array}$$

from here you could do something different to better represent the function where $x\approx y$:

Working in other question I found this other approximation: $$\ln(1+e^x) \approx \begin{cases} \frac{x}{1-e^{-\frac{x}{\ln(2)}}},\quad x\neq 0\\ \ln(2),\quad x=0\end{cases}$$

For the last one the maximum error is below $0.0092$, and the plot is quite close:

comparison of the plots

Results that applying this to your problem leads to the following approximation for splitting the logarithm of the sum of exponentials LogSumExp(x,y): $$\ln(e^x+e^y) \approx \begin{cases} \displaystyle{\frac{xe^{\frac{x}{\ln(2)}}-ye^{\frac{y}{\ln(2)}}}{e^{\frac{x}{\ln(2)}}-e^{\frac{y}{\ln(2)}}}},\quad x\neq y\\ x+\ln(2),\quad x\equiv y\end{cases}$$

it works quite good if $e^x+e^y$ is not too close to zero.

comparison of formulas

Joako
  • 1,380