Computing the gradient of quadratic form

Question

If $v$ is an $m$-vector and $X$ is an $m \times n$ full rank matrix, how can I find the value for $n$-vector $s$ that satisfies the following?

$$ \frac{\partial}{\partial s} \|Xs - v\|^2 = 0 $$

Answer 1

"Matrix differentiation" is really just a way of organizing scalar differentiation for all of the components of vectors and matrices. Here you need several results. Let $D_x$ be the gradient with respect to the vector $x$. Let $A$ be a matrix and $c$ be a constant vector. Then

1. $D_x(c) = 0$
2. $D_x(Ax) = A$
3. $D_x(x^TAx) = (A^T + A)x$

Since $\|Xs-v\|^2 = (Xs-v)^T(Xs-v) = s^TX^TXs - 2v^TXs + v^Tv$, we have

\begin{align} D_s\|Xs-v\|^2 = 2X^TXs - 2v^TX \end{align}

and therefore, if $X$ has full column rank,

$$s = (X^TX)^{-1}X^Tv.$$