16

In the proof of matrix solution of Least Square Method, I see some matrix calculus, which I have no clue. Can anyone explain to me or recommend me a good link to study this sort of matrix calculus?

In Least-Square method, we want to find such a vector $x$ such that $||Ax-b||$ is minimized.

Assume $r=Ax-b$

$\Rightarrow\|r\|^2=x^TA^TAx-2b^TAx+b^Tb$

$\Rightarrow \nabla_x \|r\|^2=2A^TAx-2A^Tb$

In the end we set the gradient to zero and find the minimized solution. I understand the whole idea, but I just don't know how exactly we did matrix calculus here, or say I don't know how to do the matrix calculus here. For example, can anyone tell me how we got those transpose in $\|r\|^2$(By what rule?) and how we got the gradient?(how do we take the gradient exactly in matrix format)?

I'll really appreciate if you can help me out. Thanks!

Rodrigo de Azevedo
  • 1
Cancan
  • 2,747

4 Answers4

11

Well the first step is the definition of $||r||^2$. This is easy \begin{align} ||r||^2 & = \langle r,r \rangle = r^T r \\ &= (Ax-b)^T(Ax-b) = (x^TA^T-b^T)(Ax-b) \\ &= x^TA^TAx -x^TA^Tb-b^TAx +b^Tb \\ &= x^TA^TAx -(b^TAx)^T -b^TAx +b^Tb \end{align} Since $(b^TAx)$ is a scalar it holds $(b^TAx)⁼ (b^TAx)^T$ Thus \begin{align} ||r||^2 & = x^TA^TAx -2b^TAx +b^Tb \end{align}

And for the derivatives, you could take a look here. Another approach would be to write out the matrix-vector expressions in sumation form and calculate the derivative, then no matrices are involved.

Thomas
  • 4,363
9

Below are the matrix/vector derivative rules, you will need.

$$\dfrac{d(x^TBx)}{d x_i} = \dfrac{d}{dx_i}\left(\sum_{j,k} x_j B_{jk}x_k\right) = \sum_{j} x_j B_{ji} + \sum_{k}B_{ik} x_k = \sum_{k}\left(B^T + B\right)_{ik}x_k$$ Hence, we have $$\dfrac{d(x^TBx)}{d x} = (B^T+B)x$$ Similarly, we have $$\dfrac{d(c^Tx)}{d x_i} = \dfrac{d}{d x_i}\left(\sum_k c_k x_k\right) = c_i$$ Hence, we have $$\dfrac{d(c^Tx)}{dx} = c$$ Now you should be able to get what you want.

3

All we need here is multivariable calculus, not matrix calculus. Let $f(x) = (1/2) \| Ax - b \|^2$. Notice that $f(x) = g(h(x))$, where $h(x) = Ax - b$ and $g(u) = (1/2) \|u \|^2$. It can easily be seen that the derivatives of $g$ and $h$ are $$ h'(x) = A, \qquad g'(u) = u^T. $$ From the multivariable chain rule, we have \begin{align*} f'(x) &= g'(h(x)) h'(x) \\ &= (Ax - b)^T A. \end{align*} If we use the convention that $\nabla f(x)$ is a column vector, then \begin{align} \nabla f(x) &= f'(x)^T \\ &= A^T (Ax - b). \end{align}

littleO
  • 51,938
1

The problem is to calculate $\nabla_xL$ given the following $$\eqalign{ r &= A\cdot x-b \cr L &= r\cdot r \cr }$$ First, calculate the differential. Then change the independent variable from $r\to x$ $$\eqalign{ dL &= 2r\cdot dr \cr &= 2r\cdot (A\cdot dx) \cr &= (2A^T\cdot r)\cdot dx \cr \nabla_xL &= (2A^T\cdot r) \cr&= 2A^T\cdot(A\cdot x-b) \cr }$$

greg
  • 35,825