Also, can someone please explain why the descent happens in a direction perpendicular to the contour lines?
Asked
Active
Viewed 216 times
2
-
Descent is performed along the gradient, perpendicular to the level sets because it has the steepest the rate of change, guaranteeing convergence to at least a local extremum if one exists. Otherwise you could just wander around the loss surface aimlessly. – Emre Jun 19 '17 at 18:05
1 Answers
1
Hinton refers to a shape like this, where clearly the horizontal cross-sections are ellipses (or circles).
(In 3D space things can be perpendicular to each other in two ways) So no matter how steep the gradient at one point is, we need to travel perpendicular (in one way) to the contour lines in order to reach the minimum of the bowl. Think about it :D
TTF
- 11
- 2