Recovering the solution of optimization problem from the dual problem

Question

In the context of (most of the times convex) optimization problems -

I understand that I can build a Lagrange dual problem and assuming I know there is strong duality (no gap) I can find the optimum of the primal problem from the one of the dual. Now I want to find the primal optimum point (i.e. the point in which the optimum is attained). Somehow, it is assumed that both the optimum of the primal and dual point are achieved at the same point (which is then a saddle point). Why/when is that true? Does strong duality can happen only at a saddle point? Does that require the primal problem to be convex?

Now, from this, it is assumed it's enough for to find the minimizer of the Lagrangian at the value of dual parameters (point of the solution of the dual problem) in order to get the primal optimal point. Why is that true?

Thank you, Dany

Answer 1

Ok, so I'm not totally sure whether this addresses all (some!) of your doubts, so let me know if it does not. As a disclaimer, what follows is just one out of several ways to view duality in optimisation (you can find others in the answers to this post).

To make things a bit concrete I'm going to assume that your primal problem is a minimisation problem with objective $f:\mathbb{R}^n\to\mathbb{R}$ and set of feasible points $X\subseteq\mathbb{R}^n$. Suppose we can find some function $\phi:\mathbb{R}^m\to\mathbb{R}$ and set $Y\subseteq\mathbb{R}^m$ that has the following property:

$\phi$ evaluated at any point $y\in Y$ gives a lower bound for $f$ over all of $X$. That is, for any $y\in Y$

$$f(x)\geq \phi(y)\quad\quad\forall x\in X.$$

Note that the above implies that for any $y\in Y$, $\phi(y)$ is also a lower bound for the optimum value of the primal (that is, $p^*:=\inf_{x\in X}f(x)$). Then an interesting question to ask is what is the best lower bound we can extract from $\phi$ and $Y$? What is $d^*:=\sup_{y\in Y}\phi(y)$?

We call answering this question "a dual problem to the original primal problem" and we say that "strong duality" holds for this primal-dual pair if $d^*=p^*$. Strong duality is handy because (assuming we can solve the dual problem) it gives us a way to check how suboptimal a candidate optimal point $\tilde{x}$ (which we have computed in some way) of the primal is by simply computing the difference $f(\tilde{x})-d^*$ (otherwise, since we don't know $p^*$, it is hard to judge how good is our candidate optimal point $\tilde{x}$). Indeed, if $f(\tilde{x})=d^*$ we know that $\tilde{x}$ is optimal and we can stop looking for a better $x$.

In order for the above to be of use, we should aim to construct a dual problem that is simple to solve. This is where the Lagrangian dual problem comes in; it is a dual problem which is always convex irrespective of whether the primal is. The objective function of the Lagrangian dual problem is

$$\phi(y):=\inf_{x\in X}L(x,y)$$

where $L$ is the so-called "Lagrangian function" and $y$ are the "Lagrangian multipliers".

To check whether strong duality holds for a given primal-dual pair there is a whole battery of sufficient conditions collectively referred to as constraint qualifications. The most basic of which applies to the case where the primal is convex and the dual is the Lagrangian dual is called Slater's condition. It asks that there exists at least one "strictly feasible" point for the primal (that is, that $X$ has non-empty relative interior). If I remember correctly, in this case strong duality holds if and only if $L(x,y)$ has a saddle point (which turns out to be a pair of optimal point $x$ and multipliers $y$), however I don't have an intuitive explanation why this is so.

With regards to how to actually find an optimal point $x^*$ what typically happens in practice is that people run an optimisation algorithm (often gradient decent or Newton's) on both (and separately) the primal and dual problems which hopefully generate some sequences $(x_n)$ and $(y_n)$ such that

$$f(x_1)\geq f(x_2)\geq f(x_3)\geq\dots,\quad\quad L(y_1)\leq L(y_2)\leq L(y_3)\leq\dots.$$

Every few iterations they stop and check whether the difference $f(x_n)-L(y_n)$ is smaller than some tolerance level and if so stop completely and say that $x_n$ optimal. They call the corresponding $y_n$ a certificate of optimality, because anyone can then verify that $x_n$ is (nearly) optimal by checking that the mentioned difference is small. This is the essence of the so called primal-dual solvers. Note that if the primal, said algorithms is guaranteed to generate $x_n$ such that $f(x_n)\to p^*$ (ditto with respect to the dual problem).

Answer 2

I know this question was asked some time ago, but I just stumbled upon it myself wondering the same thing, so I will post what I have found out in case it is helpful to someone else.

I think the answer you seek is in Boyd and Vandenberghe's Convex Optimization (freely available on Boyd's website: http://stanford.edu/~boyd/cvxbook/) in sections 5.1-5.5 (especially, see subsection 5.5.5). Below are my key takeaways from that text.

Let \begin{align} \min \;\;\;\; &f_0(x)\\ \text{s.t.} \;\;\;\; &f_i(x) \leq 0 \;\;\;\; i = 1,\ldots,m \tag{1} \label{eq:1}\\ &h_i(x) = 0 \;\;\;\; i = 1,\ldots,p, \end{align} where $x \in \mathbb{R}^n$ be the optimization problem (not necessarily convex) we're considering. The Lagrangian of this problem is $$ L(x, \lambda, \nu) = f_0(x) + \sum_{i=1}^m \lambda_i f_i(x) + \sum_{i=1}^p \nu_i h_i(x). $$ The Lagrangian dual is $$ g(\lambda,\nu) = \inf_{x \in \mathcal{D}} \left( f_0(x) + \sum_{i=1}^m \lambda_i f_i(x) + \sum_{i=1}^p \nu_i h_i(x) \right), $$ where $\mathcal{D} := \left( \bigcap_{i=0}^m \operatorname{dom} f_i \right) \cap \left( \bigcap_{i=1}^p \operatorname{dom} h_i \right)$. The Lagrangian dual problem, in turn, is \begin{align} \max \;\;\;\; &g(\lambda,\nu)\\ \text{s.t.} \;\;\;\; &\lambda \succcurlyeq 0. \end{align} Note that \begin{align} \sup_{\lambda \succcurlyeq 0, \; \nu} L(x,\lambda,\nu) &= \sup_{\lambda \succcurlyeq 0, \; \nu} \left( f_0(x) + \sum_{i=1}^m \lambda_i f_i(x) + \sum_{i=1}^p \nu_i h_i(x) \right) \\ &= \begin{cases} f_0(x) & \text{if } f_i(x) \leq 0, \; i = 1,\ldots,m, \text{ and } h_i(x) = 0, \; i=1,\ldots,p \\ \infty & \text{otherwise}. \end{cases} \end{align} This means that $$ p^\star := \inf_x \sup_{\lambda \succcurlyeq 0, \; \nu} L(x, \lambda, \nu) $$ is the optimal value of the primal problem \eqref{eq:1}. Note that there is no restriction on the $x$ values over which we take the infimum, since the conditions of \eqref{eq:1} are already included in the $\sup L$. Moreover, by definition, $$ d^\star := \sup_{\lambda \succcurlyeq, \; \nu} \inf_x L(x,\lambda,\nu) $$ is the optimal value of the dual problem.

Let us now assume that strong duality holds. Then $d^\star = p^\star$. Furthermore, also assume that both the primal and dual problem optimal values are attained, i.e., that there exist feasible $x^\star$ and $(\lambda^\star,\nu^\star)$ such that $f(x^\star) = d^\star$ and $g(\lambda^\star,\nu^\star) = d^\star$. Then \begin{align} f_0(x^\star) &= g(\lambda^\star,\nu^\star) \\ &= \inf_x \left( f_0(x) + \sum_{i=1}^m \lambda_i^\star f_i(x) + \sum_{i=1}^p \nu_i^\star h_i(x) \right) \\ &\leq f_0(x^\star) + \sum_{i=1}^m \lambda_i^\star f_i(x^\star) + \sum_{i=1}^p \nu_i^\star h_i(x^\star) \\ &\leq f_0(x^\star) \end{align} (for motivations of each equality/inequality, see page 242 in Boyd and Vandenberghe's book). This implies that the inequalities are in fact equalities. We from this, we can conclude that $x^\star$ solves the problem $\inf_x L(x,\lambda^\star,\nu^\star)$. Finally, assume that $\inf_x L(x,\lambda^\star,\nu^\star)$ has a unique solution (which happens e.g. if $L(x,\lambda^\star,\nu^\star)$ is a strinctly convex function of $x$). Then $x^\star$ is that solution, and therefore $$ x^\star = \arg\min_x L(x,\lambda^\star,\nu^\star). $$

Answer 3

If you solved the dual and have optimal values for the dual variables, then plug those optimal dual values back into the Lagrange equation. Now you have an equation with only x as unknown. Minimize, and you've recovered the value for x.