1

I am looking for the dual of the following optimization problem

\begin{alignat}{2} \tag{P1} &\underset{x \in \mathbb{R}^{n \times 1}}{\text{minimize}}&\quad \frac{1}{2}\left\|x - a \right\|_2^2\\ &\text{subject to }&\quad \|x\|_{\infty} \leq b, \end{alignat} where $a \in \mathbb{R}^{n \times 1}$ and $b \in \mathbb{R}_{\geq0}$.

Question:

How to find the dual of this problem ${\text{P}}1$ (in particular, unconstrained problem by using the dual of the infinity norm ball)?

Partial understanding (perhaps I should say everything is mixed up in my head):

As I understand so far, the dual norm of the inifinity norm ball is $\ell_1$ norm, that is, $\|x\|_1 = \underset{\|z \|_{\infty} \leq 1}{\max} z^Tx$. My "problem" is how to remove the constraint in the primal problem ${\text{P}}1$ and convert the problem as unconstrained by using the fact there is an equivalent notion of dual norm. Can anybody help me please?

Royi
  • 8,711
learning
  • 661

1 Answers1

2

Let the Lagrangian be: $$L(\mathbf{x};\lambda,\mathbf{\mu})=\frac{1}{2}\|\mathbf{x}-\mathbf{a}\|_2^2+\mathbf{\lambda}^T(\mathbf{x}-\mathbf{b})-\mathbf{\mu}^T(\mathbf{x}+\mathbf{b}),$$ where $\lambda,\mu\in \mathbb{R}^n_+$ and $\mathbf{b}=(b,b,\dots,b)$ is an $n$-long vector with all components equal to $b$. What we did is to convert the constraint $\|\mathbf{x}\|_{\infty}\leq b$ to separable linear constraints $-b\leq x_i\leq b$. The gradient of $L$ will be $$\nabla_x L = \mathbf{x}-\mathbf{a} +\lambda -\mu \overset{\nabla_x L=\mathbf{0}}{\longrightarrow}\mathbf{x}^*=\mathbf{a}-\lambda+\mu.$$ Overall we have $$L(\mathbf{x}^*;\lambda,\mathbf{\mu})=q(\lambda,\mu)=\frac{1}{2}\|\mu-\lambda\|^2+\lambda^T(\mathbf{a}+\mu-\lambda-\mathbf{b})-\mu^T(\mathbf{a}+\mu-\lambda+\mathbf{b})=\\ -\frac{1}{2}\|\mu-\lambda\|^2-\mathbf{a}^T(\mu-\lambda)-\mathbf{b}^T(\mu+\lambda)$$ and your dual problem is $$\max_{\lambda,\mu\in\mathbb{R}^n_+}q(\lambda,\mu).$$

Not the most appealing dual problem I have to admit. We moved from a one variable problem to two variables, and for iterative methods - projecting onto the $l_{\infty}$ ball is a easy as projecting onto $\mathbb{R}^n_+$. Duality comes in handy if we were able to get a problem of lower dimension or a "better" constraint set, which is not the case in hand.

iarbel84
  • 1,513
  • Thank you for your reply (+1). I agree that this dual is not appealing. On one hand, the projection approach can be considered as well. On the other hand, I was wondering if it is possible to convert the $\ell_{\infty}$ ball to $\ell_1$ (which can make the problem unconstrained). – learning Jul 31 '20 at 06:17
  • @learning first of all, I now notice that the dual problem is exactly like to primal problem. Have a good look and you'll see why. Secondly, converting a constrained problem to an unconstrained one is what we did when constructing the Lagrangian. Then the problem is converted from having just $\mathbf{x}$ as variable to having $\mathbf{x,\lambda}$. If you're looking to get an explicit solution to the problem, unfortunately that's not possible. You need an iterative method, and luckily, the problem P1 you presented is very easily and efficiently solved using first or second order methods. – iarbel84 Aug 01 '20 at 07:12
  • @learning As mentioned before, we generally use duality when it allows us to cast the problem in a way that is easier to handle by an algorithmic framework, or if it reduces the dimension of the problem. Since complexity is mostly dependent on dimension, this can provide a substantial improvement in performance. However, in your case it doesn't happen since: 1) it's easy to begin with 2) the constraint set has a dimension equal to the problem dimension, as these are box constraints (there is a separable constraint for each variable). Therefore the dual problem is of the same dimension – iarbel84 Aug 01 '20 at 07:19
  • Thank you for your reply. With respect to your first comment. Indeed, both primal and dual problems are similar. I agree with your comments. – learning Aug 02 '20 at 07:38
  • I might be missing something, but the above seems like Projection onto the $ {L}_{\infty} $ Ball. This solved in https://math.stackexchange.com/a/1825772. What am I missing? There is no need for First Order Methods, there is a closed form solution. – Royi Jan 08 '21 at 00:00