10

I'm reading a numerical analysis book, which gave me very confusing descriptions about stability and backward stability.

It says that an algorithm $f$ is stable, if for any $x$, $$ \frac{\|f_b(x)-f(x_b)\|}{\|f(x_b)\|}=\mathcal{O}(e_{machine}) $$

Because I am assuming it should be how the approximated function mapping is away from the true function, so I don't understand why it is not the $$ \frac{\|f_b(x)-f(x)\|}{\|f(x)\|}=\mathcal{O}(e_{machine}) $$

Can anyone give me some description why the stable is defined in this way, and what is the general considerations for stable analysis?

Also, for backward stability, is it true that any algorithm is stable if it is backward stable?

amWhy
  • 209,954
Jack2019
  • 1,525
  • What book? It would be helpful to know which terms are used by various authors. – Don Hatch Aug 20 '22 at 22:51
  • 1
    For reference, I'm guessing the book is Trefethen&Bau's "Numerical Linear Algebra", there's an on-line copy here and your quotes look similar to the ones on p.103, where they are given with more context explaining what the $\tilde{f}$ (your $f_b$) means. Your first formula is what they call "stable" and other authorities (Higham, wikipedia) call "mixed forward-backward stable"; your second formula is what they call "accurate" and other authorities (Higham, wikipedia) call "forward stable". – Don Hatch Aug 23 '22 at 19:25

1 Answers1

17

Conceptually (letting $\varepsilon$ be machine epsilon):

  • A method for computing some result $w$ is forward stable if the computed solution $w^\ast$ is "near" the exact solution: $\|w-w^\ast\|/\|w\|\leq c\varepsilon$, where $c$ is some not-so-large constant.

  • A method is backward stable if $w^\ast$ is the exact solution of a perturbed problem; that is, if $p$ is the problem satisfied by $w$ and $p^\ast$ is the problem satisfied by $w^\ast$, then $\|p-p^\ast\|/\|p\|\leq k\varepsilon$, where $k$ is some constant that is not too large.

The difference is in the focus: forward analysis is concerned with what the method churns out, while backward analysis looks at the problem being solved (which is why we can speak of ill-conditioned methods and ill-conditioned problems). To give a concrete example: in using Cholesky decomposition to solve a system involving the Hilbert matrix, forward analysis will blame Cholesky for the observed instability, while backward analysis will blame Hilbert. Since it turns out that Cholesky tends to solve problems with not-too-large "condition numbers" accurately, we say that Cholesky is backward stable, but not forward-stable.

Nick Higham has a more extensive discussion; you will also want to see the pioneering book by Jim Wilkinson, who came up with the concepts.

See also this other answer I gave.

Community
  • 1
J. M. ain't a mathematician
  • 75,051
  • 4
    "Life may toss us some ill-conditioned problems, but there is no good reason to settle for an unstable algorithm." - D.P. O'Leary. – J. M. ain't a mathematician Nov 04 '11 at 09:54
  • 2
    How is this answering the question? The answer is comparing forward vs backward stability whereas the question is asking purely why backward is not defined in another way? – user1559897 Feb 25 '18 at 19:49
  • 1
    @user, "why backward is not defined in another way" - because it's the useful definition involving the condition number of the problem (as espoused by Wilkinson). If you can think of a more useful way to present backward stability, have at it. – J. M. ain't a mathematician Feb 26 '18 at 04:23
  • I agree that this doesn't seem to answer the question, which is why (forward, not backward) stability is not defined in the way the asker stated. The expression the asker provided is the accuracy of the algorithm, not the stability, which are separate concepts. – kennysong Feb 16 '20 at 13:18
  • Any chance you could rephrase, or add clarification, to try to make it clear how your answer relates to the question, using the terminology used by the question? Specifically, the question is asking about a specific property "stability" with given definition, whereas your answer never mentions that word at all; instead, it appears to be changing the subject, as it talks about "forward stability" instead. If what you are calling "forward stability" is intended to be what the question calls "stability", it would greatly help to say so. – Don Hatch Aug 20 '22 at 23:07