How to find stationary values of $\frac{1}{2}x^TAx-x^Tb$?

Question

How to find stationary values of $\frac{1}{2}x^TAx-x^Tb$?

We know that we need to differentiate it with respect to $x$.

But I dont know how to differentiate?

Can someone help

Answer 1

$\newcommand{\x}{\mathbf{x}}\newcommand{\b}{\mathbf{b}}$The key facts you need are:

1) The derivative (gradient) of $\x^T A\x$ with respect to $\x$ is $\color{blue}{(A + A^T)\x}$, as shown here: Gradient of $x^{T}Ax$. Note that if $A$ is symmetric, this becomes $2A\x$.

2) The derivative (gradient) of $\x^T \b (=\b^T \x)$ with respect to $\x$ is $\b$. To see this, note that $\x^T \b = b_1x_1+b_2x_2+\cdots + b_nx _n$. So for each $i$, the partial derivative with respect to $x_i$ is $b_i$, whence the gradient is $\begin{bmatrix}b_1 \\ b_2 \\ \vdots \\ b_n\end{bmatrix} = \b$.

Answer 2

In cases such as this it is useful to fall back of the definition of differentiability. A function $f : \mathbb{R}^m \rightarrow \mathbb{R}$ is said to be differentiable at the point $x \in \mathbb{R}^m$ if there exists a vector $c \in \mathbb{R}^m$ such that $$ f(x + h) - f(x) = c\cdot h + \epsilon(h)\|h\|.$$ Here $\cdot$ denotes the Euclidian inner product, $\|x\|$ is the Euclidian norm of the vector $x$ and $\epsilon$ is a function which satisfies $$ \epsilon(h) \rightarrow 0, \quad h \rightarrow 0, \quad h \not = 0$$ While $c = \nabla f(x)$ is the only choice, the original definition can offer a short-cut as we discover now.

We have $f(x) = x^T A x - x^T b$. It follows that $$ f(x+h) - f(x) = (x+h)^TA(x+h) - (x+h)^Tb - x^TAx + x^T b = x^TAh + h^TAx + h^TAh -h^Tb = x^T(A+A^T)h-b^Th + h^TAh = [x^T(A+A^T) - b^T] h + h^TAh.$$ To verify this, it is important to use that $h^TAx = x^TA^Th$. We see that a excellent candidate for the vector $c$ is $$c = x^T(A+A^T) - b^T.$$ It remains to be seen if $$ \epsilon(h) = \frac{h^T A h}{\|h\|},$$ satisfies $$ \epsilon(h) \rightarrow 0, \quad h \rightarrow 0, \quad h \not = 0.$$ Fortunately, we have $$ \|h^TAh\| \leq \|h\| \|Ah\| \leq \|A\| \|h\|^2.$$ Here we have used Cauchy-Schartz's inequality and a key property of the matrix norm. Specifically, $$ \|Ah\| \leq \|A\| \|h\|.$$ It follows immediately that $$ \epsilon(h) \leq \|A\| \|h\| \rightarrow 0, \quad h \rightarrow 0, \quad h \not = 0.$$