Is it possible to approximate the $max\{x,y\}$ by a differentiable function? $f(x,y)=max \{x,y\} ;\ x,y>0$
Asked
Active
Viewed 9,220 times
3 Answers
17
Yes it is. One possibility is the following: Note that $\def\abs#1{\left|#1\right|}$ $$ \max\{x,y\} = \frac 12 \bigl( x+ y + \abs{x-y}\bigr), $$ take a differentiable approximation of $\abs\cdot$, for example $\def\abe{\mathop{\rm abs}\nolimits_\epsilon}$$\abe \colon \mathbb R \to \mathbb R$ for $\epsilon > 0$ given by $$ \abe(x) := \sqrt{x^2 + \epsilon}, \quad x \in \mathbb R $$ and define $\max_\epsilon \colon \mathbb R^2 \to \mathbb R$ by $$ \max\nolimits_\epsilon(x,y) := \frac 12 \bigl( x+y+\abe(x-y)\bigr). $$ Another possibility is to take a smooth mollifier $\phi_\epsilon$ and let $\max'_\epsilon :=\mathord\max * \phi_\epsilon$.
martini
- 84,101
-
(+1), good advice. One smoothing I stumbled across recently is $|x| \approx x\operatorname{erf}(ax)$ with $a \gg 1$. Taking $a=5$ works well enough. – Antonio Vargas Oct 08 '13 at 22:37
-
1So how would you approximate the following? $max(x_1,x_2,x_3)$, where $0 \le x_i \le 100$, $i=1,...,3$ ? – Clement Feb 22 '21 at 11:09
-
@Clement Perhaps, you could implement it pairwise? – Dr_Zaszuś May 20 '21 at 12:08
10
Another possibility is given by: \begin{equation} \max (x,y) \approx \ln(e^x+e^y) \end{equation} This approximation doesn't work well for similar values of $x$ and $y$. We can however, remedy this by introducing a scaling parameter $N$: \begin{equation} \max (x,y) \approx \frac1N\ln(e^{N x}+e^{N y}) \end{equation} for large values of $N$.
A general definition is given by:
\begin{equation} \max\limits_{x \in S} \approx \frac1N\ln(\sum\limits_{x\in S} e^{N x}) \end{equation}
Note that in practice $e^{Nx}$ will give unworkably large answers for large $N$.
Angelorf
- 356
- 1
- 11
-
How could deal with overflow and underflow in case of large numbers multiplied by large scaling parameter N? – Feras Oct 22 '17 at 17:49
-
-
I'm testing another function $max(x,y) = pow((pow(x,n) + pow(y,n), 1/n)$ it is doesn't have overflow but in one case when x or y is zero so I'm adding this small number 1e-5 to avoid it. Do you have any idea ? – Feras Oct 24 '17 at 16:46
-
That doesn't work for negative numbers. Why avoid zero? $0^x = 0$ (except $0^0$, but $n \neq 0$ anyway) – Angelorf Oct 25 '17 at 19:42
-
I have relu function before So I don't have any negative numbers. the problem with zero when the function is in its first derivative. – Feras Oct 25 '17 at 19:45
-
3Overflow issues can be (mostly) avoided using the identity ln(sum(exp(Nn)))/N = ln(sum(exp(N(n-m))))/N + m, where m is the actual maximum. – Ian Fellows May 18 '18 at 17:03
-
2$\max(x,x)=x\ne 0.30102+x\approx\ln{2}+\ln{e^x}=\ln(e^x+e^x)$. So, the error is huge for $x\approx y $ and near zero – Oleg Melnikov Jul 28 '18 at 15:53
2
You can use an approximation of max norm a.k.a. infinity norm with a p-norm of relatively high $p$.
Caveat #1: make sure $p$ isn't too high though because that can cause numerical instabilities, overflow to infinity, esp. if you use 32-bit floats
Caveat #2: norm works with the absolute values so you'll need to transform your numbers with e.g. log(exp(x)+1) (a.k.a. softplus) if you have to deal with negative values as well (and then back). Softplus quickly approximates $x$ for $x>20ish$ btw so this can be faster than it seems.
Tiana
- 121
- 3