Convergence of gradient descent for arbitrary convex function

Question

Suppose function $f: \mathbb{R}^d \to (-\infty, \infty]$ is convex and assume that it is differentiable over its effective domain. We do not assume that $f$ is Lipschitz, nor that it is closed. Suppose further that there is a nonempty set of points that minimize it.

Consider using gradient descent to solve the unconstrained minimization problem

\begin{align*} \min_{x} f(x) \end{align*}

with step sizes $t^k = \frac1k$.

1. Does gradient descent converge to an optimal point?

2. If it does not necessarily converge, are there any modifications we can make to the gradient descent algorithm to ensure convergence?

3. If not, what are some minimal assumptions to add to $f$ to ensure convergence?

4. If it does converge, what is the complexity?

5. Also, does it converge with backtracking line search?

Answer 1

No, gradient descent will not always converge to an optimal point. An example is given in this post.

It should be noted, first, that gradient descent is not appropriate to use with extended-real valued functions. Consider the example given in the linked post. Gradient descent will not even ensure that each successive iteration stays in $\textbf{dom}(f)$. The more appropriate method would be projected gradient descent. Moreover, projected gradient descent also fails for the example in the linked post. For example, starting at $(1,1)$, the gradient direction is always $(0, -1)$, and the projection of the point $(1,-a)$ into $D$ does not exist.

However, I believe that gradient descent with backtracking line search will always converge for the unconstrained minimization of the function $f: \mathbb{R}^d \rightarrow \mathbb{R}$ if $f$ is convex and twice continuously differentiable. The reason is that this implies that $f'$ is continuously differentiable, so it is locally Lipschitz. This means that on the set $B = \{x: \Vert x - x^* \Vert \leq x - x^0\}$, $f$ is $M$-smooth. If you perform gradient descent with constant step size $1/M$, it can be shown that each $x_k$ always stays in $B$. Thus, the problem is equivalent to the convex, smooth case, in which we have \begin{align*} f(x_k) - f(x^*) \leq \frac{2M\Vert x_0 - x^* \Vert^2}{k} \end{align*} In backtracking line search, we always have $t \leq 1/M$, so I think backtracking line search should converge as well.