1

I'm running gradient descent on a continuous function and I observe this pattern: enter image description here

What can cause such a sudden kink? Why does the Loss keep increasing after it? I understand issues related to a learning rate that is too large, but this does not seem to be the case.

The parameter also turns abruptly (it's a complex-valued parameter, or we can think of it as a pair of parameters): enter image description here

For completeness, here's the norm of the gradient (there's no large jump as if there was a cliff, see fig. 8.3 pag. 285) enter image description here

Ziofil
  • 1,590
  • What is the function that you are trying to minimize? – supinf Mar 06 '20 at 10:12
  • it's a bit complicated to write explicitly in full, but it's an inner product between a complex vector $\mathbf{u}$ and a vector $\mathbf{v}$ transformed by the parametrized unitary matrix U(z): $\mathrm{loss}(z) = -|\langle \mathbf{u}^* , U(z) \mathbf{v}\rangle|^2$ – Ziofil Mar 06 '20 at 10:22
  • There may be an error in the formulation of the gradient computation. – akkapi Jan 14 '24 at 07:39

0 Answers0