1

Show by direct estimates that if $ A \in R^{n \times n}$ , $A > 0$ and $ b \in R^n$ then the function $$\frac{1}{2}\langle Ax,x\rangle - \langle b,x\rangle$$ with $x$ is convex on $R^n$.

My approach: A function $g : i \rightarrow R$ is said to be convex if $g(tx + (1-t)y) \le tg(x) + (1-t)g(y)$, $\forall x,y$ in $i$, and $0 \le t \le 1$

Hence, $g(x) = \frac{1}{2}\langle Ax,x\rangle - \langle b,x\rangle$ with $x$, $\Rightarrow g(tx + (1-t)y) = \frac{1}{2}\langle A[(tx+(1-t)y)], tx + (1-t)y\rangle - \langle b, tx + (1-t)y\rangle$.

From here I got stuck expanding the term, any help is highly appreciated.

user2323232
  • 672
Ilan Aizelman WS
  • 4,128
  • 1
    I would use the fact that $\langle Ax, x \rangle = |\sqrt{A} x|^2$, where $\sqrt{A}$ is the unique positive definite square root of $A$. – Theo Bendit Apr 23 '19 at 12:31
  • @TheoBendit Could you please show further how could it be used please? – Ilan Aizelman WS Apr 23 '19 at 12:32
  • 1
    From there, convexity should follow from the definition. How comfortable are you with expanding inner products and square-norms? Also, are you able to show that $f(x) = x^2$ is a convex function over $[0, \infty)$? It might help with this question. – Theo Bendit Apr 23 '19 at 12:39

1 Answers1

2

Since you're not comfortable showing $f(x) = x^2$ is convex, let's begin with that. Naturally, you can prove this by showing that $f''(x) \ge 0$ for all $x$, but we want to prove it by definition.

Suppose $x, y \in \Bbb{R}$ and $\lambda \in [0, 1]$. Then \begin{align*} &\lambda f(x) + (1 - \lambda) f(y) - f(\lambda x + (1 - \lambda)y) \\ = \; &\lambda x^2 + (1 - \lambda)y^2 - (\lambda x + (1 - \lambda)y)^2 \\ = \; &\lambda x^2 + (1 - \lambda)y^2 - \lambda^2 x^2 - 2\lambda x(1 - \lambda) y - (1 - \lambda)^2 y^2 \\ = \; &\lambda(1 - \lambda)x^2 + \lambda(1 - \lambda)y^2 - 2\lambda(1 - \lambda)xy \\ = \; &\lambda(1 - \lambda)(x^2 - 2xy + y^2) \\ = \; &\lambda(1 - \lambda)(x - y)^2 \ge 0. \end{align*} Thus, $$\lambda f(x) + (1 - \lambda) f(y) \ge f(\lambda x + (1 - \lambda)y),$$ and $f$ is convex.

Now, suppose $x, y \in \Bbb{R}^n$ and $\lambda \in [0, 1]$. By linearity of $\sqrt{A}$, triangle inequality, and positive scalar homogeneity of the norm respectively, \begin{align*} \|\sqrt{A}(\lambda x + (1 - \lambda)y)\| &= \|\lambda \sqrt{A} x + (1 - \lambda) \sqrt{A} y\| \\ &\le \|\lambda \sqrt{A} x\| + \|(1 - \lambda) \sqrt{A} y\| \\ &= \lambda\|\sqrt{A} x\| + (1 - \lambda) \|\sqrt{A} y\|. \end{align*} Thus the map $x \mapsto \| \sqrt{A} x\|$ is also convex. Using this and the previous inequality, \begin{align*} &\lambda\|\sqrt{A} x\|^2 + (1 - \lambda)\|\sqrt{A} y\|^2 - \|\sqrt{A}(\lambda x + (1 - \lambda)y)\|^2 \\ \ge \; &\lambda\|\sqrt{A} x\|^2 + (1 - \lambda)\|\sqrt{A} y\|^2 - (\lambda\|\sqrt{A} x\| + (1 - \lambda)\|\sqrt{A} y\|)^2 \\ = \; &\lambda(1 - \lambda)(\|\sqrt{A} x\| - \|\sqrt{A} y\|)^2 \ge 0. \end{align*} Therefore, the map $x \mapsto \|\sqrt{A} x\|^2$ is convex, as required. Use the fact that $$\|\sqrt{A} x\|^2 = \langle \sqrt{A} x, \sqrt{A} x\rangle = \langle\sqrt{A} \sqrt{A} x, x\rangle = \langle A x, x\rangle,$$ as $\sqrt{A}$ is positive and hence Hermitian.

Theo Bendit
  • 50,900
  • I see, how can I continue from here to show that $\frac{1}{2}\langle Ax,x\rangle - \langle b,x\rangle$ is convex? – Ilan Aizelman WS Apr 23 '19 at 13:27
  • 1
    Note that $x \mapsto \langle -b, x\rangle$ is a linear and hence convex function. The sum of two convex functions is convex, and is straightforward to prove. – Theo Bendit Apr 23 '19 at 13:29
  • @TheoBendit why can you say that $A^{1/2}x,A^{1/2}x>=<A^{1/2}A^{1/2}x,x>$ if it’s not hermitian matrix? – user2323232 Apr 26 '19 at 07:59
  • @user2323232 $A^{\frac{1}{2}}$ is Hermitian. It is defined to be the unique positive definite matrix (which, inclusive into the definition, means its Hermitian) $B$ such that $B^2 = A$. – Theo Bendit Apr 26 '19 at 09:35
  • @TheoBendit is there any source saying that positive matrix on $R$ should be hermitian? I don’t really remember this part – user2323232 Apr 26 '19 at 09:37
  • @user2323232 Like I said, it's part of the standard definition. From Wikipedia: "... a complex $n \times n$ Hermitian matrix $M$ is said to be positive definite if the scalar $z^*Mz$ is strictly positive for every non-zero column vector $z$ of $n$ complex numbers." I'm not sure what definition you prefer, but this is the definition I've been using, and it's perfectly standard. – Theo Bendit Apr 26 '19 at 09:42
  • @TheoBendit for example $A=\begin{bmatrix}1&1\-1&1\end{bmatrix}$ and $x=[a \space b]$ then $<Ax,x>=a^2-ab+ba+b^2=a^2+b^2>0$ in contradiction, am I wrong? – user2323232 Apr 26 '19 at 13:40
  • 2
    @user2323232 It's not a contradiction, it's a matter of definition. $A$ is not positive-definite because it only satisfies one of the two conditions that make up positive-definiteness. Specifically, it fails to be Hermitian. (Also, $\langle Ax, x \rangle$ may not be a real number, let alone positive, when looking at complex vectors.) Here's some extra reading: https://math.stackexchange.com/questions/1107230/why-is-positive-semi-definite-only-defined-for-symmetric-matrices and https://math.stackexchange.com/questions/267300/positive-definite-matrix-must-be-hermitian – Theo Bendit Apr 26 '19 at 14:49
  • @TheoBendit another question, your proof also shows that this function is strictly convex am I right? – user2323232 May 08 '19 at 08:10
  • @user2323232 Yes, that's right. – Theo Bendit May 08 '19 at 08:20