Are there any Theoretical Results that Illustrate why Convex Optimization is Generally Easier than Non-Convex Optimization?
Although this is a straightforward idea to understand - I am trying to find some mathematical results that show why a Convex Function is generally easier to optimize compared to a Non-Convex Function. For example, suppose the same optimization algorithm (e.g. Gradient Descent) is tasked with optimizing a Convex Function and a Non-Convex Function of similar complexity (e.g. highest exponential power in both functions is "n").
My Question: In such a case - how could we show that the optimization algorithm is likely to "struggle more" with the Non-Convex Function compared to the Convex Function?
The first thing that comes to mind is that Non-Convex Functions can possess "saddle points" whereas Convex Functions can not possess "saddle points". "Saddle Points" are points on the function being optimized where the second derivatives of the function are 0, but "saddle points" are not the minimum of the function. "Saddle Points" present themselves as an obstacle where optimization algorithms can get "stuck" - logically speaking, we understand this by thinking that the second derivatives of the function at a "saddle point" are 0, since the direction in which the optimization algorithm travels depends on the second derivatives at each point until successive differences are negligible, "saddle points" will trick the algorithm into believing a minimum has been reached because future evaluations of the second derivative at "saddle points" will remain as 0. But even though this is a logical and common argument, I am not sure if this serves as a "proof" that Non-Convex Functions, by virtue of the fact that they can possess "saddle points", are generally more difficult to optimize compared to Convex Functions.
The second thing that comes to mind is that perhaps some theoretical convergence results might have been established that show "stronger rates of convergence" (e.g. Pointwise vs. Uniform, Landau Notation, Linear Time vs. Polynomial Time, etc.) for Gradient Descent on Convex Functions compared to Non-Convex Functions - but my knowledge of mathematics is too weak to understand and decipher such results if they exist . If such results existed, then we could have an easier time arguing that Convex Optimization is generally easier than Non-Convex Optimization (when comparing the same optimization algorithm on functions with similar complexity).
Can someone please comment on this? What kind of theoretical results exist to suggest that Convex Optimization is generally easier than Non-Convex Optimization?
Thanks!
