I often read that deep learning suffers from a lack of theory, compared to classical machine learning. I mean that deep learning has shown to be a powerful tool in practice but there is no proof of this effect in theory. Which leads to my question: Is there some conjectures in deep learning theory? What should be proven mathematically to build a real deep learning theory?
Asked
Active
Viewed 378 times
10
-
It would be helpful when asking about a popularized phrase (buzzword) to offer your own definition or at least narrow the Question by providing a link to a definition you accept. Wikipedia provides one such treatment, but the description "deep learning" is often applied to architectures as well as "techniques" (algorithms). Some foundation is needed to offer an informed opinion as to whether a topic "suffers from a lack of theory". – hardmath Oct 12 '18 at 16:56
1 Answers
1
I don't know of any conjecture that captures all theoretical problems raised by deep learning . Also, there is not much consensus on how to properly formalize the questions. For example Naftali Tishby's group uses an information theoretic approach , while others - like Sanjeev Arora ,looks more to the (non-convex) optimization problems and different simplified or specific architectures (random or linear networks).
A good theory of deep learning should explain two things:
1.Why they generalize so well ,although the standard learning theory indicate that they should overfit .
2.Why the optimization is practically so easy ,although the current theory say it is hard.
Popescu Claudiu
- 547