I can not find out when to use positive or negative Lagrange multipliers. Does it depend on if I am looking for MAX or MIN ? or maybe it depends on inequality I mean if it is bigger than zero or lower than zero ? I have a feeling that I get right results no matter what is the sign but in one case they hold for conditions and in the another doesnt ?
Thank you
It certainly does make a difference.
In the case Karl talks about, you may not care as it deals with an equality constraint. But when dealing with inequality constraints, it is crucial. If you're having trouble memorizing if it's supposed to be a plus or a minus in front of the multiplier, just think about what you want to be penalizing. For example
$\min f(x) \\@ \ Ax \leq b$
with a Lagrangian of $L(x,\mu) = f(x) \pm \mu(Ax-b)$. You want to punish $Ax -b > 0$. If there's a minus sign in front of the multiplier and $Ax-b <0$, then $L(x,\mu)$ actually decreases. So you're improving the Lagrangian objective function by violating a constraint. This is the exact opposite of what you want to do. That's all you need to keep in mind. Just give it whatever sign conforms with what you want to achieve, that is violations of constraints should act against what you want to achieve.
edit: Also keep in mind that you can reformulate any problem to be of some "canonical" form in which you have memorized the sign. $\max f(x), Ax \geq b$ can be reformulated as $\min -f(x), -Ax \leq -b$ etc. If you want to memorize something, memorize it for ONE form then transform all other instances to this form.
The ambiguity comes from the fact that you can write the Lagrangian as: $L(\mathbf{x},\lambda)=f(\mathbf{x})+\lambda g(\mathbf{x})$ or $L(\mathbf{x},\lambda)=f(\mathbf{x})-\lambda g(\mathbf{x})$. Both lead to the same solutions, but the signs of the Lagrange multipliers are different. See also:
