1

I was trying to take the gradient of $x^TAx$ i.e. $\nabla_xx^TAx$.

I did have one idea of how to do this which was expression $x^TAx$ as a double summation and then take the partial derivatives wrt to each $x_i$. However, it seemed a little ugly to me, and therefore, was wondering if there was a different maybe more cleaver/cleaner way to derive it.

The kind of idea I was thinking to apply was maybe the product rule of gradients:

$$\nabla_x(uF) = \nabla_xuF + u\nabla_xF$$

Where u is a scalar function and F is a vector field. (I got the above from the following OCW video).

However, I was not 100% sure how to apply it. The issue I had was that if I let $u = x^T$ and $Ax = F$, then F does correspond to a vector field because to start with, it is a valid vector. However, $u = x^T$ does not correspond correctly to a scalar function because its a vector. I guess maybe I am not using the most general version of the product rule? Is there a more general version of it such that it yields a nicer derivation for $\nabla_xx^TAx$?

Charlie Parker
  • 6,546
  • Found a related question: http://math.stackexchange.com/questions/482742/how-to-calculate-gradient-of-xtax?rq=1 – Charlie Parker Nov 23 '14 at 02:06
  • Please feel free to leave different derivations, if you think you have a clean alternative answer (even if the question already has a accepted answer and you think you can still provide a nice/clean derivation for it!) :) – Charlie Parker Nov 23 '14 at 03:08
  • Check out the resources on matrix differentials listed at the bottom of this question: https://math.stackexchange.com/questions/4105372/clean-simple-way-of-computing-nabla-fu-u-mathsft-with-respect-to-u/4105679#4105679 – kanso37 Apr 18 '21 at 03:31

1 Answers1

7

$$J(x) = x^TAx = \sum_{i,j} A_{ij}x_ix_j \implies \dfrac{dJ(x)}{dx_k} = \sum_{i,j} A_{ij} \dfrac{d(x_ix_j)}{dx_k}$$ We have $$\dfrac{d(x_ix_j)}{dx_k} = \delta_{ik}x_j + \delta_{jk}x_i$$ Hence, \begin{align} \dfrac{dJ(x)}{dx_k} & = \sum_{i,j} A_{ij} \left(\delta_{ik}x_j + \delta_{jk}x_i\right) = \sum_{j}A_{kj}x_j + \sum_{i}A_{ik}x_i = \left((Ax)_k + (A^Tx)_k\right) \end{align} Hence, $$\dfrac{dJ}{dx} = \left(A+A^T\right)x$$

Adhvaitha
  • 5,441
  • Very nicely done. – GDumphart Nov 23 '14 at 01:44
  • Do you mind explaining a little more you second line with the deltas? What do the deltas exactly mean? Is it possible for a little more detail there? – Charlie Parker Nov 23 '14 at 02:30
  • 1
    @Pinocchio $\delta_{ij}$ is the Kronecker delta, i.e., $\delta_{ij} = 0$ if $i \neq j$ and $\delta_{ij}=1$ if $i=j$. – Adhvaitha Nov 23 '14 at 02:32
  • For the last line when you use the subscript k to "index" from a vector, is that standard notation? Or should you explicitly say a small comment instead of just saying "Hence". Personally, it took me a little while to see why the step was correct, not sure if it was notational and wanted to make it more useful for everyone else too just to avoid confusion. Btw, thanks a lot and a very clean way of deriving it btw! :) – Charlie Parker Nov 23 '14 at 03:07