Proof of ADMM special case

Question

I was reading the textbook about ADMM and came across some special cases without proof, which is really interesting so I was trying to prove them. I did a couple but had some trouble getting this one so please help. Thank you in advance

Considering x-update when A=I

$x^+=argmin_x(f(x)+(ρ/2)∣∣x−v∣∣_2^2)=prox_{f,ρ}(v)$.

The special case to be proved is $f(x)=(1/2)x^TPx+q^Tx+r, x^+:= (P+{\rho}A^TA)^{-1}({\rho}A^Tv-q)$

The attached photo shows what I have tried so far, I think I am close but there is something not right.

Are you familiar with matrix calculus? If not, I would recommend to do so such that you can derive these cases by yourself. — user550103, Dec 04 '20 at 06:45
What exactly is your question and what have you tried so far? — iarbel84, Dec 04 '20 at 14:21
I have edited my question and showed what I have tried, sorry for the unclarity. — slamdunk, Dec 05 '20 at 05:52
Computing the gradient of the function $h(x) = x^T Ax$ is just plain old vector calculus, even though a matrix is involved. — littleO, Dec 05 '20 at 09:20
I am bit confused, what exactly do you want to prove? Do you want to obtain this update $x^{+} = \arg\min_{x} \frac{1}{2} x^TPx + q^Tx + r + \frac{\rho}{2}| x - v |_2^2$? — user550103, Dec 05 '20 at 10:44

score 0 · Accepted Answer · answered Dec 06 '20 at 10:11

Hint (almost full solution)

Assuming you want to solve the following update

\begin{align} x^{+} = \arg\min_{x} \frac{1}{2} x^TPx + q^Tx + r + \frac{\rho}{2}\| Ax - b - v \|_2^2. \end{align}

Compute the gradient and set it to zero to obtain the $x$-update solution.

Gradient of $\frac{1}{2} x^TPx + q^Tx+ r$ is $\frac{1}{2}(P+P^T)x + q$ (see example here How to take the gradient of the quadratic form?). Assuming $P$ is symmetric, the gradient simplifies to $Px + q$.
Gradient of the last piece, i.e., $\frac{\rho}{2}\|Ax-b-v\|^2$, is $\rho A^T(Ax-b-v)$.

To this end, add the gradients and set them to zero such that \begin{align} Px + q + \rho A^T(Ax-b-v) = 0 \end{align}

Can you take it from here?

Yes, that is very clear and straightforward, thank you. – slamdunk Dec 06 '20 at 22:17 — slamdunk, Dec 06 '20 at 22:17

Proof of ADMM special case

1 Answers1