8

I'm just learning about optimization, and the difference between an analytical solution, and a numerical one. Suppose there is a cost function f(x), and we want to find the value of x which minimizes this. In an analytical solution, we would differentiate with respect to x, i.e. find df(x)/dx. Whereas in a numerical solution, we would try values of x, and see in which direction we need to change x in order to move towards the solution (gradient descent).

It seems to me that the analytical solution is always preferred because it gives you an exact answer. You do not get stuck in local minima and you can be sure that your solution is correct. So why do we not always use analytical solutions?

I understand that numerical solutions make things easier because you don't have to work out by hand the analytical solution, but it seems to me that solving something by hand to get a precise answer is highly preferred to doing it computationally for an approximate answer. Can somebody clear this up for me? Thanks!

J. M. ain't a mathematician
  • 75,051
Karnivaurus
  • 827
  • 2
    How do you find the roots of $f'(x)$ analytically for sufficiently weird functions? – Hagen von Eitzen Sep 14 '15 at 13:41
  • 12
    Because you can't always obtain an analytic solution. It's really that simple. – Michael Grant Sep 14 '15 at 13:41
  • 1
    In some of the cases, the analytical solution may not be numerically as stable as calculating the solution numerically. For example, calculating inv(A) * B can be difficult in case det(A) is very small. On the other hand, solving Ax = B using numerical methods can provide more stable approximations to the true solution. – musically_ut Sep 14 '15 at 20:05
  • Find a general analytic solution to Navier-Stokes and I'll happily use it instead of numerical solving. – casey Sep 15 '15 at 00:14

4 Answers4

11
  • Some equations have no finitely expressible analytic solution ($x^5+x+1=0$, for example).
  • Symbolic algebraic manipulation is computationally expensive, even when it can produce a usable solution.
  • For some functions, even taking the derivative analytically is too difficult.
  • You don't always need an exact solution: sometimes you just want bounds on the answer.
Chappers
  • 67,606
  • 5
    Roots of $x^5+x+1$ can't be expressed in terms of arithmetic and radicals, but saying it doesn't have an analytical solution is probably unreasonable. It's solution is certainly more elementary than that of, say, $\log(x) = 2$, it just not expressed by a common token. –  Sep 14 '15 at 17:17
  • 1
    That's what I meant when I said "finitely expressible". Maybe it's less clear than I first thought. – Chappers Sep 14 '15 at 17:35
  • 3
    Roots of the quintic are analytically and finitely expressible using elliptic integrals. See http://math.stackexchange.com/questions/540964/how-to-solve-fifth-degree-equations-by-elliptic-functions for more. – Eric Towers Sep 15 '15 at 05:05
9

Well, I would agree that an analytical solution generally is preferable, if one can find it. The problem is that in many, many problems, finding analytical solutions is very difficult or even impossible. These problems can still often be solved by numerical methods.

J. M. ain't a mathematician
  • 75,051
Eff
  • 12,989
  • 5
    Preferable for what? Which is more useful: $−2.2582588834026085818$ or $$\frac{-\sqrt [3]{756+12,\sqrt {3873}}}{6}-{\frac {4}{\sqrt [3]{756+12, \sqrt {3873}}}}$$ Useful for what? – GEdgar Sep 14 '15 at 14:05
2

Numerical solutions are quick and dirty:

Quick

This means faster than the analytical way, possible in human life time or even just possible.

Dirty

This means not exact like analytical ones and often enough not too wrong.

If you have a system which can correct things during the flight, you can even fly things to comets far away.

Another reason:

Often, our input data has errors, so exact computation is somehow not as good anymore.

Community
  • 1
Antitheos
  • 801
  • 2
    While they are not exact it is usually possible to get a numerical solution of arbitrary precision. If you need a better result, it just takes more time. – Josef Sep 14 '15 at 15:18
2

In addition to answers already given, it might well be that your "analytic expression" will have to be numerically evaluated in applications anyway, and that its evaluation will often be less efficient, more numerically unstable, or even both, than had you just constructed a numerical solution at the outset.

Let me quote the following from Acton, who puts it more eloquently than I ever could:

Formal mathematical training is a frequent if minor villain in the righteous struggle for efficient computer use. For example, every college sophomore has seen linear algebraic equation solving expressed in two lines as $$\begin{align*}&\qquad\mathbf A\mathbf x=\mathbf b\\&\text{so}\\&\qquad\mathbf x=\mathbf A^{-1}\mathbf b\end{align*}$$ It is not surprising that, faced with an equation set, he invokes a computer library program to invert his matrix $\mathbf A$ and then multiply it into $\mathbf b$. The fact that this sequence of operations takes at least three times as much labor on the part of the computer as a direct solution of the original problem by the elimination of variable technique has never penetrated his consciousness. The glibness of the formal mathematical notation has obscured the realities of the arithmetic process.

...

The student that worships at the altars of Classical Mathematics should really be warned that his rites frequently have quite oblique connections with the external world.

and in a later section

$$F(b)=\frac{\pi}{2}\cos b-\left[\cos b\int_0^b\frac{\sin t}{t}\mathrm dt+\sin b\int_b^\infty\frac{\cos t}{t}\mathrm dt\right]\qquad\qquad\text{(10.17)}$$ This formulation of our function is another way to get the series representation. Since the two integrals are functions tabled in several standard references [AMS 55] $\text{(10.17)}$ is also the practical formula for hand evaluation of $F$. When we consider automatic computation, however, this expression for $F(b)$ is virtually useless, since it requires the subcomputation of two nonstandard functions, one of them containing a singularity at the origin. We quote it here merely to stress how different in utility a particular expression may be for hand and automatic computations.

J. M. ain't a mathematician
  • 75,051