Minimum value of $f(x) = x^{T}Ax$ when $A$ is symmetric and positive definite

Question

So if $A$ is symmetric positive definite, how can we show that the graph of:

$f(x) = x^{T}Ax$ is concave up. Furthermore, what is the minimum value of this function?

Answer 1

First observe that given $x,y$ and $\lambda\in [0,1]$ you have \begin{align*} f(\lambda x+(1-\lambda)y) &= (\lambda x+(1-\lambda)y)^T A (\lambda x+(1-\lambda)y)\\ &= \lambda^2 x^T A x + (1-\lambda)^2 y^TAy +\lambda (1-\lambda) x^T A y + \lambda (1-\lambda) y^T A x\\ \end{align*} Now the last two terms can be bounded in a nice way, indeed since $A$ is positive definite, you have $0\leq (x-y)^T A (x-y) = x^T A x + y^T A y - x^TAy-y^TAx$ so that $x^TAy+y^TAx\leq x^TAx+y^TAy$. Inserting that in the above equation you get \begin{align*} f(\lambda x+(1-\lambda)y) &\leq \lambda^2 x^T A x + (1-\lambda)^2 y^TAy +\lambda (1-\lambda) x^T A x + \lambda (1-\lambda) y^T A y\\ &= \lambda x^T A x+(1-\lambda) y^TAy\\ &= \lambda f(x) +(1-\lambda) f(y) \end{align*} So that $f$ is convex.

If you look at $g(x) = \frac{1}{2}x^{T}Ax - x^{T}b$, we know it is convex since it is basically $1/2 \cdot f$ plus something linear. So to minimize it, it is enough to set the gradient to $0$. The gradient is \begin{align*} \nabla g(x) &= \frac{1}{2} (A^T x+Ax)-b\\ &= Ax-b \end{align*} This is of course equal to $0$ when $x=A^{-1}b$.

Answer 2

Concave up actually means that the function is convex. We can check if a function is convex by looking at the hessian. $$\ f(x) = x^TAx \\ H = 2A $$ Since, the hessian is psd. We can conclude that the function is convex. For convex functions minimizer is given by $\nabla f(x^*) = 0$. For convex functions local minimizer is global minimizer.
Hope this helps.