2

This is an idea that I had while playing with an automatic differentiation system built on dual numbers. This system, like most computer algebra systems built on floating point arithmetic, has the extra values "inf," "-inf," and "NaN," which means we are basically working with the extended real numbers. This system also has an element called $\epsilon$ with $\epsilon^2 = 0$, thus implementing the dual numbers.

For whatever reason, we have that the reciprocal of $1/\epsilon$ in this system is equal to "$\text{inf} - \text{inf} \epsilon$". As a result, we have that $\epsilon/\epsilon = \text{NaN} + \text{NaN} \epsilon$. This can occasionally get in the way of things, particularly when trying to evaluate huge compositions of functions with something like an implicit $x/x$ term in there.

This problem can be solved if we instead let the reciprocal of $\epsilon$, let's call it $\omega$, be an "infinite-potent" quantity that squares (and cubes, etc) to "inf", and with $\omega \epsilon = 1$. Now, if I want to see what happens to $\sin(x)/x$ as $x \to 0$, I simply put in $\sin(\epsilon)/\epsilon$ and get the value $\epsilon/\epsilon = 1$, the correct answer. We have extended "automatic differentiation" to a kind of "automatic L'Hôpital's rule."

Question: This technique is not only elegant but seems to be incredibly useful in computations. I cannot possibly be the first person to think of it. All we are doing is taking the idea behind extended real numbers, and building up an "extended dual numbers" with "infinite-potent" elements that square (or cube, etc) to $\infty$ as reciprocals of nilpotent elements. The result extends automatic differentation. Does this have a name or literature on it?

To be clear, the structure I am talking about is all elements of the form $a \omega + b + c \epsilon$, along with the special elements $\infty$ (and maybe $-\infty$), with $\epsilon^2 = 0$ and $\omega^2 = \infty$. Of course you could also have $\epsilon^3 = 0$ and $\omega^3 = \infty$ instead, etc.

Mike Battaglia
  • 7,878
  • This sounds like Robinson's Non-Standard Analysis, where you consider an extended real line $\hat{\mathbb R}$ that has, besides finite numbers, infinite and infinitesimal numbers that satisfy the condition 1/infinitesimal=infinite. However, in $\hat{\mathbb R}$ infinitesimals don't square to zero, because if in a ring you have an invertible nilpotent element, then $1=0$ – Alessandro Oct 08 '22 at 06:49
  • I am familiar with nonstandard analysis but am asking about something different here. Not sure what you mean by your last point, of course the extended real numbers aren't a ring... – Mike Battaglia Oct 08 '22 at 07:08
  • 1
    I do not believe that a procedure of this kind can possibly produce consistent results. How do you propose to distinguish $e^{-\infty} = 0$ from $e^{\infty} = \infty$ this way? – Qiaochu Yuan Oct 08 '22 at 07:51
  • This structure reminds me of the projective line over the ring of dual numbers, studied by J. Grünwald in 1906 (see the Wikipedia page). However, there we have $\infty=-\infty$ and $\omega \epsilon$ is undefined. Note that trying to impose $1=\omega \epsilon$ leads to inconsistent results if one wants to preserve usual arithmetic laws, since multiplying by $\epsilon$ on both sides one arrives at $\epsilon=\omega \varepsilon^2=\omega 0 = 0$. – pregunton Oct 08 '22 at 10:35
  • For one approach using nilpotent infinitesimals see the links here to Synthetic Differential Geometry. – Bill Dubuque Oct 08 '22 at 11:32
  • @pregunton thanks for the reference, that is very close to what I was looking for. Yes, saying that $\omega \epsilon = 1$ means the resulting structure is non-associative. I guess you are suggesting that makes things a little less nice from an abstract algebra standpoint. If so I agree with that although I am still curious if the result is useful as a way to handle inverses of nilpotents in computer algebra systems. – Mike Battaglia Oct 08 '22 at 16:59
  • To clarify my question: if we are going to have values like "inf" and "-inf" at all in our computer algebra system, as reciprocals of zero, then does it make sense for a square root of zero to have a reciprocal that is a square root of infinity? This seems to have useful computational properties wrt automatic differentiation. It clearly is non-associative, but is it alright beyond that, or is there some kind of dealbreaker level of logical inconsistency it introduces? – Mike Battaglia Oct 08 '22 at 17:02
  • @QiaochuYuan do I understand your comment correctly as being an objection to the extended real numbers in general, rather than the structure I am suggesting? FWIW, in the computer algebra system I am talking about, we currently have $\exp(1/\epsilon)=1/\epsilon=\infty−\infty \epsilon$ and $\exp(−1/\epsilon)=0+\text{NaN} \epsilon$; not sure if that's better or worse than what I was suggesting. – Mike Battaglia Oct 08 '22 at 17:19
  • You can extend the real numbers however you want but I do not believe it is possible to write down an algebraic structure which automatically evaluates limits, except for a very restrictive class of functions (rational functions are clear, I'm not sure what else beyond that). – Qiaochu Yuan Oct 08 '22 at 17:21
  • Can you elaborate on how this would not work? It seems to give the right result for $\sin(x)/x$, and a slight generalization to higher-order nilpotents would give the right result for $\sin(x^n)/x^n$ for whatever $n$, and if we really want to do things properly we could ditch nilpotents entirely and use the Levi-Civita field or etc, which seems like it'd give the right results for everything. Or would it? – Mike Battaglia Oct 08 '22 at 17:35
  • If $\omega\epsilon=1$ then you sacrifice associativity because $(\omega\epsilon)\epsilon\ne \omega(\epsilon\epsilon)$. – Anixx Oct 17 '22 at 13:52
  • See here https://mathoverflow.net/questions/432396/extending-reals-with-logarithm-of-zero-properties-and-reference-request how one can extend dual numbers to make sense of $\epsilon^\epsilon$ using divergent integrals. By adding another element $\omega=\int_0^\infty dx$ one can make sense of $\ln \epsilon$ as well (it would be $\lambda+\epsilon \omega$. Buy I am afraid to add $1/\epsilon$ one has to make sense of $1/0$ first. – Anixx Oct 17 '22 at 14:02
  • Hi @Anixx, the thing about associativity was talked about above... I don't think it's much of a problem for a computer algebra system as those are typically non-associative anyway. I will respond to the rest on the other post... – Mike Battaglia Oct 18 '22 at 17:41

0 Answers0