2

This is a question that I have always been interested in:

Suppose we have a system of equations - for example, a system of non-linear equations. From what I have seen, in most cases, we tend to use numerical optimization methods to attempt and find an approximate solution. The idea being, we simply assume that an analytical solution to this system of non-linear equations probably does not exist, and some numerical approximation technique (e.g. Stochastic Gradient Descent) will likely be able to provide a suitable answer in an acceptable amount of time.

But this brings me to my question. Consider the case of Neural Networks (Machine Learning) - in Neural Networks, we are required to find an "optimal" set of weights that minimizes the Loss Function of the Neural Network. Doing this is effectively attempting to "solve" a system of non-linear equations.

It has almost become routine to "solve" these systems of non-linear equations using numerical optimization methods such as "Stochastic Gradient Descent", as we implicitly assume that there probably does not exist an analytical solution to this system of non-linear equations. Although in theory there "might" exist an analytical solution to this system of non-linear equations "perhaps if we looked hard enough" - we just "assume" it probably doesn't exist or if it did exist it would take too much work to uncover, and thus we "solve" this system of equations numerically and "hope" that the numerical solution is "good enough".

But can we actually prove this: Can we mathematically demonstrate that some specific system of non-linear equations does not have an analytical solution?

The closest mathematical theorem I could find that addresses this point is the Abel-Ruffini Theorem (https://en.wikipedia.org/wiki/Abel%E2%80%93Ruffini_theorem) - but I am not sure if this theorem is specifically making claims about the question I am asking.

  • Can someone please comment on this : For optimization problems in which we rely on numerical approximations - is it possible that some of these problems might actually have analytical solutions?

Thanks!

stats_noob
  • 3,112
  • 4
  • 10
  • 36
  • 3
    Where is the implicit assumption? If there is an analytical solution, then numerical approximation methods will still work just as well as always. – Misha Lavrov Jul 20 '22 at 02:41
  • @ Misha: thank you for your comment! I mean the implicit assumption - in Neural Networks, why do we use Stochastic Gradient Descent instead of trying to do matrix/algebra manipulation and see if some analytical solution exists? The obvious answer is because this would take too much time, the solution might not exist, and a numerical approximation provides a good enough answer in a reasonable amount of time. But can we mathematically prove that an analytical solution exists or doesn't exist for a specific system of non-linear equations? Thank you so much! – stats_noob Jul 20 '22 at 02:53
  • 2
    If we have a good numerical solution, maybe we don't care whether there is an analytical solution. We know there's an analytical solution for quartic equations, but if we need a solution of a quartic to, say, five decimals, we use the numerical methods anyway, they're much simpler/quicker. And there are situations where we can prove there are no analytical solutions, such as in finding $\int_0^1\sqrt{x^3+x+1},dx$ – there is a whole theory of "integration in finite terms" that tells us when an antiderivative can/cannot be found. – Gerry Myerson Jul 20 '22 at 02:57
  • 1
    (1) I think that the loss minimization in ML is not equivalent to solving equations -- it's more an optimization problem.

    (2) For your "non-analytic" solution, it depends on the details like what you exactly mean be non-analytic. Maybe this is related? https://math.stackexchange.com/questions/982984/prove-that-an-equation-has-no-elementary-solution

     – Peter Franek Jul 20 '22 at 08:43
  • @ Peter: thank you for your reply! I was just curious in knowing if its possible to mathematically prove whether a closed form solution exists for a given system of equations. Thank you so much! – stats_noob Jul 20 '22 at 16:20
  • 1
    @ Gerry: thank you for your reply! Thats a good point - maybe it doesnt matter and we dont care! was just curious in knowing if its possible to mathematically prove whether a closed form solution exists for a given system of equations. Thank you so much! – stats_noob Jul 20 '22 at 16:21
  • 1
    What is an "analytical solution"? Does $x^2-2=0$, $x\in[1,2]$ have an analytical solution? If you reduce a polynomial system to a triangular system, so that every solution is obtained from a chain of univariate polynomials, does that count as "analytical" solution? Does provably finding a point in the basin of quadratic convergence count as "analytical" solution? Or do you only allow symbolic expressions? – Lutz Lehmann Jul 23 '22 at 10:26
  • 1
    To build upon previous comment: there is no clear-cut limit to what is defined as "analytical". It is always possible to give a name to a function that is defined by a system of equations. There is quite a lot of them, check "special functions": https://en.wikipedia.org/wiki/Special_functions – Jean-Armand Moroni Jul 24 '22 at 16:43

2 Answers2

1

Let $f:A\to B$ be a mapping, then an “equation” can be written as $f(a)-b=0$ with $a \in A$ and $b\in B.$ From here, you can wind up in almost every branch of mathematics.

With neural networks, $A$ is usually the set of C-infinity functions whose domain is a vector space (for example, the set of all possible 1080p images) and range is some kind of decision space (for example, a binary set of images that contain a hot dog or not hot dog). $B$ would be the positive reals, representing the error between a “solution” (the neural network and it’s learned weights) and the true function that you are trying to model. Note that we can solve this equation numerically with any number of models; neural networks are just a popular choice. Check out Rasmussen’s book GPML for an alternative. Sometimes, good old linear regression works best.

Some people don’t even try to find an analytical solutions to whatever it is they’re throwing neural networks at because the tools are becoming so easy to use. Some are successful with this approach! There are libraries out there that even automatically identify which parts of the input space ($A$ from my example above) are important and how they interact most significantly that a physicist might use to analytically find the function that the neural network is mimicking.

There are many kinds of equations for which an analytical solution exists. The implicit function theorem (and the many cool extensions) comes to mind as particularly fertile ground to guarantee the existence of analytical solutions to systems of equations.

There are classes of systems of equations for which we know no analytical solution exists.

You can analytically solve optimization problems to get solutions to systems of equations, by the way, instead of chucking them at neural networks. In my notation above, we usually try to find some function $g$ such that $|f-g|$ is minimized using gradients and Hessians and stuff like that. Check out Ruszczynski’s book “Nonlinear Optimization.”

Charles F
  • 696
  • @ Charles F: Thank you for your answer! – stats_noob Jul 23 '22 at 04:23
  • I forgot to mention - sometimes finding that an equation has no analytical solution is really cool. Robert Bryant owns the greatest answer of all time along these lines: https://mathoverflow.net/questions/171733/is-it-possible-to-express-int-sqrtx-sqrtx-sqrtx1dx-in-elementary-fun – Charles F Jul 23 '22 at 04:39
1

When you cannot solve a problem by hand, an analytical solution does not exist. Numerical solution by the aid of a computer or a calculator could be possible, if an answer exists.

Z Ahmed
  • 43,235
  • 1
    @ Z. Ahmed: thank you for your comment! But how am I supposed to truly know that I can not solve this problem by hand - or if this problem is actually solvable by hand, and I am just not able to do this? Thanks! – stats_noob Jul 23 '22 at 05:41