0

Suppose that there is a positive definite matrix $\mathbf{A} \in \mathbb{R}^{n \times n}$, and a vector $\mathbf{b} \in \mathbb{R}^{n}$, then minimization of quadratic functions with linear terms can be done in closed form as $$\arg\min_{\mathbf{x} \in \mathbb{R}_n } \left( \frac{1}{2}\mathbf{x}^\mathsf{T} \mathbf{A} \mathbf{x} - \mathbf{b}^\mathsf{T}\mathbf{x} \right) = \mathbf{A}^{-1} \mathbf{b}$$

I met this in a machine learning book. However, the book didn't provide a proof. I wonder why this can be well-formed. Hope that someone can help me with it. I find that many machine learning books like to skip all of the proofs, which made me uncomfortable.

Rodrigo de Azevedo
  • 1
UESTCfresh
  • 622
  • The minimisation of which quadratic forms? What's the actual equation you're attemting to solve? That said, I am fairly certain I have seen this on Wiki and that it uses some straightforward first and second differentiation techniques – FShrike Apr 10 '22 at 08:24
  • Start with the symmetry of $\bf A$. Then use this. – Rodrigo de Azevedo Apr 10 '22 at 08:38

1 Answers1

3

One way of proving this is to "complete the square": $$ \frac{1}{2}\mathbf{x^\top A x - b^\top x}=\frac{1}{2}\left( \mathbf{\big(x-A^\mathrm{-1}b\big)^\top A \big(x-A^\mathrm{-1}b\big) - b^\top A^\mathrm{-1}b}\right)\ . $$ Because $\ \mathbf{A}\ $ is positive definite this is never less than $\ {-}\frac{1}{2}\,\mathbf{b^\top A^\mathrm{-1}b}\ $, and it attains that value when $\ \mathbf{x= A^\mathrm{-1}b}\ $.

lonza leggiera
  • 28,646
  • 2
  • 12
  • 33