5

I just came across the following

$$\nabla x^TAx = 2Ax$$

which seems like as good of a guess as any, but it certainly wasn't discussed in either my linear algebra class or my multivariable calculus class. Is there any intuitive way to see why this should be true?

Rodrigo de Azevedo
  • 1
Benjamin Lindqvist
  • 3,112

6 Answers6

5

Try it first with a simple case, such as $\nabla x^Tx$. Well, if we have

$$ x=(x_1,\ldots,x_n), $$then

$$ x^Tx=\sum_{i=1}^n x_i^2. $$ It follows

$$\nabla x^Tx = 2(x_1,\ldots,x_n)=2x $$

Now, how would you write $x^TAx$?

Edit: If $A$ is not symmetric, you can do something similar and derive

$$\nabla x^TAx=(A+A^T)x$$

Philip Hoskins
  • 2,055
2

It's only true if $A$ is symmetric. And as for intuition, consider the one-dimensional case: the derivative of $ax^2$ is $2ax$. I always recommend to write out the quadratic form and calculate the derivative by hand. Once you've done that, you'll understand and you'll never forget it anymore.

Matt L.
  • 10,636
2

This is true only if $A$ is symmetric.

For example, if $A^T=-A$, then $x^TAx=0$, and hence $\nabla x^TAx=0\ne 2Ax$.

A way to see this is $$ (x+h)^TA(x+h)-x^TAx=x^TAh+h^TAx+{\mathcal O}(|h|^2)=2(Ax,h)+{\mathcal O}(|h|^2). $$

Yiorgos S. Smyrlis
  • 83,933
1

I don't know of any intuitive method. But I would like to give you the following generalization:

$$\nabla x^TAx = (A+A^T)x$$

Basically, work with $$f(x) = \sum_{i=1}^n\sum_{j=1}^nx_i a_{ij}x_j$$

and use $a_{ij}= a_{ji}$

Gautam Shenoy
  • 10,318
0

I assume A is quadratic(otherwise the term in your question does not hold)
If you know that:

  • any symmetric matrix $A$ can be decomposed to $A=LU$

  • $\nabla Ax=A$ [and $\nabla x^TA = \nabla(A^Tx)^T=(\nabla A^Tx)^T=(A^T)^T=A$]

  • The product rule holds: $\nabla uv=(\nabla u)v+u\nabla v$

You can write:

$$\nabla x^TAx \\ =\nabla x^TLUx \\ =(\nabla x^TL) Ux + x^TL \nabla Ux \\ =LUx+x^TLU \\ =Ax + x^TA \\ =Ax + x^TA^T \\ =Ax + Ax\\ =2Ax \\ $$

user2740
  • 319
0

Let's write the $i$th component. Here, I use Einstein's convention, where a sum is assumed for repeated indices. What follows assumes that $A$ is constant.

$\displaystyle \left[ \vec{\nabla}\left( x^T A x \right)\right]_i = \partial_i \left( x_j A_{jk} x_k\right) = \left(\partial_i x_j\right) A_{jk} x_k + x_j A_{jk} \left( \partial_i x_k\right) = \delta_{ij} A_{jk} x_k + x_j A_{jk} \delta_{ik} = A_{ik} x_k + x_j A_{ji}.$

Therefore,

$\displaystyle \vec{\nabla}\left( x^T A x \right) = A x + x^T A$.

If $A$ is symmetric, then

$\displaystyle A x + x^T A = 2 A x$.

jordix
  • 631