Proximal Operator of $ \alpha \| \mu \|_{\infty} + I_{0 \leq \mu \leq 1} ( \mu ) $ Using the Moreau Decomposition

Question

I need to solve the following problem: $$\min._{0 \le \mu \le 1} \|\mu-z\|_2^2 + \alpha\|\mu\|_\infty$$ I indicate the box constraint on $\mu$ explicitly as a function and get $$\min._{\mu\in\mathbb{R^n}} \|\mu-z\|_2^2 + \alpha\|\mu\|_\infty + I_C(0 \le \mu \le 1)$$

Hence, my optimization problem is about computing the proximal operator of the function $\alpha\|\mu\|_\infty + I_C(0 \le \mu \le 1)$. Inspired by L Infinity ($ {L}_{\infty} $) Norm Regularized Proximity Problem (Or The Proximal Operator of the $ {L}_{\infty} $ (Infinity Norm)), I thought I will use Moreau's decomposition: $I = \text{prox}_f(.) + \text{prox}_{f^*}(.)$

In order to derive the conjugate of the sum of two functions, I used the concept of infimal convolution given in Dimitri P. Bertsekas - Convex Optimization Theory - Chapter 3 - Exercises and Solutions: Extended Version.

Subsequently, I am able to derive closed-form expressions for the proximal operator of the conjugate of the sum of functions and obtain the optimal $\mu$.

In the above notes, the conjugate of the sum of functions, is defined as the closure of the infimal convolution of the conjugates of the individual functions, provided the original sum of functions is proper. What is the practical consequence of dealing with the concept of closure?

Answer 1

The problem is given by the explicit Proximal Operator:

$$\begin{aligned} \arg \min_{x} \quad & \frac{1}{2} {\left\| x - y \right\|}_{2}^{2} + \alpha {\left\| x \right\|}_{\infty} + {I}_{0 \preceq \cdot \preceq 1} \left( x \right) \end{aligned}$$

Looking at the Lagrangian:

$$ L \left( x , {\lambda}_{1}, {\lambda}_{2} \right) = \frac{1}{2} {\left\| x - y \right\|}_{2}^{2} + \alpha {\left\| x \right\|}_{\infty} + {\lambda}_{1}^{T} \left( x - \boldsymbol{1} \right) - {\lambda}_{2}^{T} x $$

The KKT Conditions are given by (See the Sub Gradient of the $ {L}_{\infty} $ Norm):

$$ \begin{align*} \nabla L \left( x, {\lambda}_{1}, {\lambda}_{2} \right) = x - y + \alpha \operatorname{sign} \left( {x}_{j} \right) \boldsymbol{e}_{j} + {\lambda}_{1} - {\lambda}_{2} & = 0 && \text{(1) Stationary Point} \\ {\lambda}_{1}^{T} \left( x - \boldsymbol{1} \right) & = 0 && \text{(2) Slackness} \\ {\lambda}_{2}^{T} x & = 0 && \text{(3) Slackness} \\ x & \preceq 1 && \text{(4) Primal Feasibility} \\ -x & \preceq 0 && \text{(5) Primal Feasibility} \\ {\lambda}_{1} & \geq 0 && \text{(6) Dual Feasibility} \\ {\lambda}_{2} & \geq 0 && \text{(7) Dual Feasibility} \end{align*} $$

One need to carefully analyze the different cases of $ y $ and $ x $.

I tried to do a short cut.
Borrowing the idea from Proximal Operator for $ g \left( x \right) = \mu { \left\| x \right\| }_{1} + {I}_{{\left\| x \right\|}_{2} \leq 1} \left( x \right) $ ($ {L}_{1} $ Norm and Unit Ball Constraint) I thought the solution would be a composition of the Proximal Operator of the $ {L}_{\infty} $ Norm and Projection onto the Box Constraint.

I tried once with doing first the $ \operatorname{prox}_{\alpha {\left\| \cdot \right\|}_{\infty}} \left( \cdot \right) $ (See The Proximal Operator of the $ {L}_{\infty} $ (Infinity Norm)) and then $ \operatorname{Proj}_{\boldsymbol{0} \preceq \cdot \preceq \boldsymbol{1}} \left( \cdot \right) $ yet found cases it didn't work and then tried the other way around and still founds cases it didn't work.

The solution I found is doing both and setting the solution to the one which yield lower objective value.

To verify the result I implemented it in MATALB and verified it against CVX.
The code is available at my StackExchange Mathematics Q2301266 GitHub Repository.

Answer 2

When I was facing this problem, I was working on a formulation in which I had to compute the proximal operator of the $L_{\infty}$ norm of a vector, with the elements of the vector being subject to simplex constraints. While working on this problem, I tried a formulation, where I replaced the simplex constraint with box constraints between $0$ and $1$. Later, I realized that simplex constraint is better for me.

Subsequently, I also changed my formulation in order to consider the simplex constraint as a quadratic barrier term in the objective function itself. Then I came to know that such a formulation has already been tried. For example, you could refer this paper: Scalable Exemplar Clustering and Facility Location via Augmented Block Coordinate Descent with Column Generation.

Answer 3

The closure is not needed here. A bit more details, given functions $f_1,\dots,f_n$, since the original sum of functions is proper, we know that \begin{equation} \text{ri}~(\text{dom}~f_1) \cap \dots \cap \text{ri}~(\text{dom}~f_n) \neq \emptyset. \end{equation} In this sense, the closure is not necessary. You can refer to Rockafaller's book ``Convex Analysis'' (Theorem 16.4) for more details.

For this problem, it is equivalent to \begin{equation} \begin{array}{cl} {\min_{\mu, \beta}} & {\|\mu-z\|^2 + \alpha \beta} \\ {\text{s.t.}} & {-\beta \leq \mu \leq \beta} \\ {} & {0 \leq \mu \leq 1} \end{array} \end{equation} The result is the lower lower objective value of setting $\beta = 1$ and setting $0 < \beta < 1$.

1. For $\beta=1$, we can easily obtain that optima $\mu^* = \operatorname{proj}_{0 \leq \mu \leq 1}(z)$, i.e., \begin{equation} \mu_i^* = \left\{ \begin{array}{cl} {0} & {z_i \leq 0}\\ {z_i} & {0 < z_i < 1} \\ {1} & {z_i \ge 1} \end{array} \right. \end{equation}

2. For $0 < \beta < 1$, the problem can be rewritten as \begin{equation} \begin{array}{cl} {\min_{\mu, \beta}} & {\|\mu-z\|^2 + \alpha \beta} \\ {\text{s.t.}} & {0 \leq \mu \leq \beta} \\ \end{array} \end{equation} Thus the closed-form solution of $\mu$ is $\mu^* = \operatorname{proj}_{0 \leq \mu \leq \beta}(z)$, i.e., \begin{equation} \mu_i^* = \left\{ \begin{array}{cl} {0} & {z_i \leq 0}\\ {z_i} & {0 < z_i < \beta} \\ {\beta} & {z_i \ge \beta} \end{array} \right. \end{equation} Then we need to solve the optimization problem in term of $\beta$: \begin{equation} \begin{array}{cl} {\min_{\beta}} & {(z)_{-}^2 + (z - \beta)_{+}^2 + \alpha \beta} \end{array} \end{equation} where $(x)_{-} = \min(x, 0)$ and $x_{+}= \max(x, 0)$. Wlog, suppose that the elements of the vector are in descending order, i.e., $z_1 \ge z_2 \ge \dots \ge z_n$. Let the number of elements in $z$ which are greater than $\beta$ be $m$, and take the gradient be vanished, we have \begin{equation} \sum_{i=1}^m (\beta - z_i) + \alpha = 0. \end{equation} Thus \begin{equation} \beta = \frac{\sum_{i=1}^m z_i - \alpha}{m}. \end{equation} And since there are only $m$ elements in $z$ which are greater than $\beta$, we need \begin{equation} z_m > \beta > z_{m+1}, \end{equation} That is \begin{equation} \begin{aligned} m\cdot z_m &< \sum_{i=1}^m z_i - \alpha, \\ m \cdot z_{m+1} &> \sum_{i=1}^m z_i - \alpha. \end{aligned} \end{equation}