7

As I understand it, the method of Lagrangian multipliers follows the form

Minimize $f(x,y)$ subject to constraint $g(x,y)= c$

and involves an equation of the form

$L(x,y,\lambda) = f(x,y) + \lambda (g(x,y) -c)$.

Does it matter whether I add or subtract $\lambda$? That is, is

$L(x,y,\lambda) = f(x,y) - \lambda (g(x,y) -c)$

equally valid? Why or why not? Is this different if I am doing a maximization problem?

Stan Shunpike
  • 4,921
  • Unless....note that $g(x,y) = c$, so that term is really zero. And it doesn't matter whether you add or subtract zero. Identity element. So then it wouldn't matter. – Stan Shunpike Jan 10 '15 at 22:00
  • the same because the theorem states that if f is ... than there exists lambda s.t ... so in your case we tkae -lambda from the original statement – Snoopy Jan 10 '15 at 22:01
  • Careful: The sign does matter when there are not only equality constraints $g(x)=0$ but also inequality constraints $g(x)\geq0$. The best explanation I've found so far is Appendix E, Bishop, Pattern Recognition. – lucidbrot Jan 28 '22 at 21:23

1 Answers1

9

It doesn't matter. All that changes is the sign of $\lambda^*$, where $(x^*,y^*,\lambda^*)$ is the critical point. Dealing with maximization doesn't change it either. You can see this because regardless of how you formulate the method, you still have

$$\nabla L(x,y,\lambda) = \begin{bmatrix} f_x(x,y) \pm \lambda g_x(x,y) \\ f_y(x,y) \pm \lambda g_y(x,y) \\ \pm (g(x,y)-c) \end{bmatrix} = 0.$$

Some advanced material follows. It is not really useful for Calc III, but it is useful if you're interested in deeper work in optimization.

There are advantages to picking one of the conventions. For instance, consider the case when $g$ is convex (and not constant) and we are minimizing $f$. Then $A=\{ x : g(x) \leq c \}$ is connected and bounded, with $\partial A=\{ x : g(x)=c \}$. Then the minimum of $f$ on $\partial A$ occurs at the minimum of $L(x,\lambda)=f(x)-\lambda (g(x)-c)$ over $A \times [0,\infty)$. We can see this because for $(x,\lambda) \in A \times [0,\infty)$, the term $-\lambda (g(x)-c)$ is a penalty for being inside $A$ rather than on $\partial A$. It's a penalty because it makes $L$ bigger and we are minimizing $L$.

The opposite convention is natural with a maximization problem with a concave constraint function $g$: the maximum of $f$ on $\partial A$ will occur at the maximum of $L(x,\lambda)=f(x)+\lambda (g(x)-c)$ over $A \times [0,\infty)$.

This perspective is useful; it is the origin of the class of numerical optimization methods called interior point methods. It also suggests various physical interpretations of the Lagrange multipliers themselves, such as "shadow prices" in economics.

Ian
  • 101,645
  • 1
    Hi, I understand this question has been a while but I have a question regarding picking the conventions. It seems to me the convention is better to be reversed if we think from the 'penalty' perspective. For example, to minimize $f$, $\lambda(g(x)-c)$ should be a penalty to make $f$ bigger when $g(x)-c > 0$ (i.e. penalize when constraint is violated). Since $\lambda > 0$, the function is better to pick plus than minus: $L(x, \lambda) = f(x) + \lambda(g(x)-c)$. – user2830451 Jan 24 '18 at 19:33
  • @user2830451 The first case is a minimization with convex constraint. Thus the interior of the domain is where $g \leq c$. If $\lambda >0$ and you are inside the domain then $-\lambda(g(x)-c)>0$ which makes the Lagrangian bigger which is bad for minimization. From this point of view the exterior simply does not exist. – Ian Jan 24 '18 at 20:05
  • 1
    Thank you so much for replying, but I don't quite understand what you said. I mean, I understand the constraint is $g(x) - c \leq 0$ for minimize $f$, but isn't it equivalently means we want $L(x, \lambda)$ bigger when the constraint is violated, i.e. when $g(x)-c>0$? So if $L(x,\lambda)=f(x)+\lambda(g(x)-c)$ and $\lambda$ is positive, it can 'penalize' $L$ when the constraint is violated. I think the point I'm confused is, isn't from the penalty perspective, we shall consider the value when the constraint is violated? (I assume 'violated' in this context is what you mean by exterior?) – user2830451 Jan 24 '18 at 23:07
  • @user2830451 If the only way to violate the constraint is to go inside the domain, then the only way to break it is to have $g(x)-c<0$. If you go outside the domain, then you have $g(x)-c>0$, but there is no reason to allow that. – Ian Jan 25 '18 at 00:40