Matrix Differentiation Chain Rule

Question

I want to compute the derivative of a scalar with respect to a matrix where the scalar is linked with the matrix through a vector. For instance let the scalar be $l = (x-c)^T (x-c)$ where $x,c$ are vectors $\in \mathbb{R}^p $ And $x = Az$ where A is a $pxq$ matrix.

Now if I want to calculate $\frac{d l}{d A}$ then I can write $\frac{d l}{d A} =\frac{d l}{d x} \frac{d x}{d A}$

$\frac{d l}{d x}$ is easy to compute but I need help for $\frac{d x}{d A}$

You don't want to do that. $\frac{\partial x}{\partial A}$ is some 3 dimensional array. Your best bet is to write $l$ in terms of $A$ and directly take the derivative from there. — Ninad Munshi, Oct 22 '19 at 11:30
So, you want to compute the gradient $\nabla_A | A z - c |_2^2$? — Rodrigo de Azevedo, Oct 22 '19 at 12:48
One method is to compute the directional derivative and massage things till something that looks like an inner product comes out. Take a look at this example. Yes, the gradient is with respect to a vector rather than a matrix, but the idea is the same. — Rodrigo de Azevedo, Oct 22 '19 at 13:15

greg · Accepted Answer · 2020-10-16T12:23:40.380

Define the vector $w=(Az-c)\;$ and write the function as $\,\ell=\|w\|^2$
Then calculate the gradient (wrt $w$) and change variables from $\,w\to A$ $$\eqalign{ \frac{\partial\ell}{\partial w} &= 2w \\ d\ell &= 2w:dw \;=\; 2w:dA\,z \;=\; 2wz^T:dA \\ \frac{\partial\ell}{\partial A} &= 2wz^T \\ }$$ where a colon denotes the Frobenius inner product, i.e. $$A:B = {\rm Tr}(A^TB) = {\rm Tr}(B^TA) = B:A$$

NB: By using the differential $(d\ell)$ in the intermediate step, there was no need to calculate the third-order tensor $\left(\frac{\partial x}{\partial A}\right)$ which is hard to calculate and awkward to manipulate algebraically.

Matrix Differentiation Chain Rule

1 Answers1