Is the Hessian of quadratic functions always symmetric?

Question

If we have

$$ f(x) := \frac{1}{2}x^\top A x - bx $$

then its gradient is $ \nabla f(x) = Ax - b $ and its Hessian is $A$. The function and its derivatives are all continuous, but $A$ (the Hessian) can be non-symmetric. What am I missing?

Answer 1

Actually the hessian of this function is ${1\over 2}(A+A^T)$ not $A$, which is always symmetric as opposed to $A$. An easy proof is to differentiate $${1\over 2}x^TAx={1\over 2}\sum_{i,j}x_ix_ja_{ij}$$w.r.t. $x_k$ and $x_l$ and note that terms like $a_{kl}+a_{lk}\over 2$ begin to emerge.

Answer 2

Let's check it with an example. Let's say:

$$A= \begin{bmatrix} 1 & 2\\ 3 & 4 \end{bmatrix}$$

Let's put $b=0$. Then you have $(x,y) A (x,y)^T=x^2 + 4y^2 +5xy$, which obviously has a symmetric hessian. So, you have indeed something wrong, and you can see what it is by using the definition of differentiability. Taking $h$ "small" (let me be informal here, you can do everything formally if you want) you have that the derivative along $h$ of your function is:

$$\lim_{h \to 0} \frac{(x+h)A (x+h)^T -xAx^T}{||h||}= e_h (A + A^T)x$$

So, in fact, the gradient of your function was $\frac{1}{2}(A+ A^T)x -b$, and if you derive again you obtain the hessian $\frac{1}{2}(A+A^T)$. Going back to my example, you can check by yourself it works.