So I'm learning about optimization of quadratic forms and this textbook goes through definiteness of matrices and principle minors etc. and then goes straight onto optimizing with constraints but never mentions how to solve the general problem of finding stationary points on $f(x)$ where $$f(x)=x^TAx + \mathbf b^Tx+c$$
$A$ being an n x n matrix and b an n x 1
Any help is appreciated
Let $\vec\nabla f$ be a $n \times 1$ vector, we will call this $\dfrac{\partial f}{\partial x}$. Then we get that $$\dfrac{\partial f}{\partial x} = Ax + A^Tx + b.$$ The differentiation with respect to a vector is obtained by recalling the following vector differentiation identities.
If $y \in \mathbb{R}^{m \times 1}$, $x \in \mathbb{R}^{n \times 1}$, then $\dfrac{\partial y^T}{\partial x}$ is a $n \times m$ matrix $A$ with $A(i,j) = \dfrac{\partial y_j}{\partial x_i}$.
Also, the chain rule goes as follows. $$\dfrac{\partial (\cdot)}{\partial x} = \sum_k \dfrac{\partial y_k^T}{\partial x}\dfrac{\partial (\cdot)}{\partial y_k}$$ Hence, we have the following relations $$\dfrac{\partial x^T}{\partial x }= I_{n \times n}$$ $$\dfrac{\partial (c^Tx)}{\partial x} = \dfrac{\partial (x^Tc)}{\partial x} = c$$
Hence, $$\dfrac{\partial b^Tx}{\partial x} = b$$ and \begin{align} \dfrac{\partial \left(y_1(x)^TAy_2(x) \right)}{\partial x} & = \dfrac{\partial y_1^T}{\partial x}\dfrac{\partial \left(y_1(x)^TAy_2(x) \right)}{\partial y_1} + \dfrac{\partial y_2^T}{\partial x}\dfrac{\partial \left(y_1(x)^TAy_2(x) \right)}{\partial y_2}\\ & = \dfrac{\partial y_1^T}{\partial x} \dfrac{\partial \left(y_2(x)^TA^Ty_1(x) \right)}{\partial y_1} + \dfrac{\partial y_2^T}{\partial x} \dfrac{\partial \left(y_1(x)^TAy_2(x) \right)}{\partial y_2}\\ & = \dfrac{\partial y_1^T}{\partial x} \dfrac{\partial \left((Ay_2(x))^Ty_1(x) \right)}{\partial y_1} + \dfrac{\partial y_2^T}{\partial x} \dfrac{\partial \left((A^Ty_1(x))^Ty_2(x) \right)}{\partial y_2}\\ & = \dfrac{\partial y_1^T}{\partial x} Ay_2(x) + \dfrac{\partial y_2^T}{\partial x} A^Ty_1(x) \end{align} In our case, $y_1(x) = y_2(x) = x$ and hence $\dfrac{\partial y_1^T}{\partial x} = \dfrac{\partial y_2^T}{\partial x} = I_{n \times n}$. Hence, $$\dfrac{\partial (x^TAx)}{\partial x} = I A x + I A^T x = \left(A+A^T \right)x$$ Setting the gradient to zero, we get the equation $$(A+A^T)x = -b$$ Solve this linear system to get the stationary points.
The unconstrained optimization problem can be formulated as
$x^{*}=\arg\min_{x}\quad f(x)$
Thus, the unconstrained Lagrangian is $L=f(x)$, which not contains lagrangian multipliers. Then, according to the KKT necessary conditions for optimallity
$\nabla_{x}L=x^{*,T}(A+A^{T})+b^{T}=0$
Solving for $x^{*}$
$x^{*}=-(A+A^{T})^{-1}b$
Finally, just verify that the Hessian is positive definite ,i.e.$(A+A^{T})>0$, to ensure sufficient optimallity conditions.
