Computing the derivative of $\|Ax\|_2$

Question

Compute the following derivative (in matrix form) $$\frac{\partial\, \|Ax\|_2}{\partial x}$$ where $A$ is an arbitrary matrix and $x$ is a vector.

I think somebody said that the result is $2A^TAx$, but I cannot get even there. I have no idea how to develop this norm and derive it in matrix form. Because if it were $$\frac{\partial \|x\|}{\partial x}=\frac{\partial x^Tx}{\partial x}=x$$ but with that matrix in the middle. I do not know how to solve that. Thanks in advance.

The gradient of the square of the Euclidean norm of $Ax$, $| Ax |_{2}^{2}$, is $2A^{T}Ax$. — Brian Borchers, Mar 30 '19 at 13:45
How do you define $\partial/(\partial x)$ where $x$ is a vector? — , Mar 30 '19 at 13:48
Compute the directional derivative. Then extract the gradient. Take a look at this. — Rodrigo de Azevedo, Mar 30 '19 at 14:01
Or alternatively take a look at this: https://math.stackexchange.com/a/2890663/550103 — user550103, Mar 30 '19 at 14:14

Traws · Answer 1 · 2019-03-30T16:26:45.293

It is useful to introduce the Frobenius inner product as:

$$ A:B = \operatorname{tr}(A^TB)$$

with the following properties derivied from the underlying trace function

$$\eqalign{A:BC &= B^TA:C\cr &= AC^T:B\cr &= A^T:(BC)^T\cr &= BC:A \cr } $$

Then we work with differentials to find the gradient. Your problem becomes, with $u=Ax$

$$\eqalign{ f &= \|u\|_{F}^{2} = u:u \\ df &= 2u : du\\ &= 2Ax : A dx\\ &= 2A^TAx : dx} $$

Thus

$$ \frac{\partial f}{\partial x} = 2 A^TAx$$

EDIT:

For $g = \|u\|_{2} $ this becomes:

$$\eqalign{ g&= f^{1/2} \\ dg &= \frac{1}{2} f^{-\frac{1}{2}} : df\\ dg &= \frac{1}{2} f^{-\frac{1}{2}} : 2u : du\\ &=f^{-\frac{1}{2}} u : du\\ &=\frac{1}{||Ax||} Ax : Adx\\ &=\frac{1}{||Ax||} A^TAx : dx\\ }$$

$$\frac{\partial g}{\partial x}= \frac{1}{||Ax||}A^T A x$$

Nice answer! Alienflow also asked for $\frac{\partial g}{\partial x}$ where $g(x) = ||A x ||$. Could you add $\frac{\partial g}{\partial x}= \frac1{||Ax||}A^T A x$ to your answer? — irchans, Mar 30 '19 at 14:08

mvw · Accepted Answer · 2019-03-30T14:43:06.973

In the Euclidean norm ($p$ norm for $p=2$) $$ \lVert y \rVert = \left( \sum_i y_i^2 \right)^{1/2} $$ for the $k$–th coordinate of the gradient by applying the chain rule several times we have $$ \begin{align} \partial_k \lVert Ax \rVert &= \frac{\partial}{\partial x_k} \left( \sum_i \left( \sum_j a_{ij}x_j \right)^2 \right)^{1/2} \\ &= \frac{1}{2 \lVert Ax\rVert} \sum_i 2 \left( \sum_j a_{ij}x_j \right) a_{ij} \delta_{jk} \\ &= \frac{1}{\lVert Ax\rVert} \sum_i a_{ik} \left( \sum_j a_{ij}x_j \right) \\ &= \frac{1}{\lVert Ax\rVert}\left( A^T A x \right)_k \end{align} $$ Note:Funny enough here I was able to recycle another answer from this morning.

Computing the derivative of $\|Ax\|_2$

2 Answers2