Basic matrix calculus question

Question

I am struggling with a basic matrix calculus question.

Suppose you had $f(\textbf{x}) = (\textbf{a} - \textbf{Bx})^T(\textbf{a} - \textbf{Bx})$

where $\textbf{a}$ and $\textbf{B}$ are a vector and matrix of constants respectively. I am interested in finding the gradient of $f$ with respect to $\textbf{x}$. I would appreciate it if someone could write down the steps needed to compute this gradient.

I know that $d(\textbf{a} - \textbf{Bx})/d\textbf{x} = -\textbf{B}^T$, but I don't know how to apply the derivative operator to the product of $(\textbf{a} - \textbf{Bx})$ with itself.

EDIT: the gradient and the derivative are not the same thing. I am interested only in the gradient.

You could also try applying the product rule from regular single variable calculus — information_interchange, Feb 05 '20 at 16:24

greg · Answer 1 · 2020-02-05T17:30:14.200

0

Recall that transposing a scalar leaves it unchanged.
Then consider the scalar function $$\eqalign{ f &= y^Tz \\ df &= dy^Tz + y^Tdz \quad &({\rm product\, rule}) \\ &= z^Tdy + y^Tdz \quad &({\rm transposed\, first\, term}) \\ }$$ Now set $z=y\,$ to obtain $$\eqalign{ df &= 2y^Tdy \\ }$$ Now let $y=(Bx-a)\,$ and therefore $\,dy=B\,dx$ $$\eqalign{ df &= 2y^TB\,dx = (2B^Ty)^T\,dx \\ }$$ So the gradient is $$\eqalign{ \frac{\partial f}{\partial x} &= (2B^Ty) = 2B^T(Bx-a) \\ }$$

edited Feb 05 '20 at 17:30

answered Feb 05 '20 at 16:13

greg

35,825

In your second equation, why is $df = y^Tdz + z^Tdy$ true? – Sam Feb 05 '20 at 16:27
I have annotated the first few steps. Better? – greg Feb 05 '20 at 17:32
Makes sense, thanks – Sam Feb 05 '20 at 20:14

Basic matrix calculus question

1 Answers1