Eigendecomposition factors relationship

Question

We know that the eigendecomposition of a square matrix $C\in\mathbb{R}^{n\times n}$, yields the following-

$$C=V\Lambda V^T$$

where

$V\in\mathbb{R}^{n\times n}$ is the matrix of eigen-vectors of $C$

$\Lambda\in\mathbb{R}^{n\times n}$ is the diagonal matrix containing eigen-values of $C$

Since the eigen-vectors happen to form an orthonormal basis, the product $V^TV=I_n$. This lets us conclude that $V^T$ is the left inverse of $V$. I have seen canonically $V^T$ being written interchangeably with $V^{-1}$, indicating that it's the right inverse of $V$ as well.

$$VV^T=\sum_{i=1}^nv_iv_i^T=I_n$$

where $v_i$ is the i$^{\text{th}}$ column vector of $V$ or eigen-vector of $C$.

I don't see why the above condition has to always hold true.

$v_i^Tv_i=1,~v_i^Tv_j=0:i\neq j$ immediately yields $V^TV=I_n$, but I'm struggling to see why this implies that the sum of their outer-products also has to equal the Identity matrix.

I can prove it mathematically using $Cv_i=\lambda_iv_i\implies CV=V\Lambda\implies V^{-1}=V^T$, but was looking for a proof that ties it to the orthonormality of the eigen-vectors.

Answer 1

$$ A = \left[\begin{array}\ a_1^T\\ a_2^T\\ \vdots \\ a_n^T \end{array}\right] $$ $$ B = \left[\begin{array}\ b_1& b_2& \cdots & b_n \end{array}\right] $$

$$ AB = \left[\begin{array}\ a_1^Tb_1 & a_1^Tb_2 & \cdots \\ &\\ &\ddots \\ a_n^Tb_1 & a_n^Tb_2 & \cdots \end{array}\right]\qquad(1) $$

So in your $V^TV$, the diagonal $a_i^Tb_i$ are ones and remaining elements are zeros. Thus forming an identity matrix.

$AB = I$ and so $B^TA^T = I$.

Now consider, $$A^TB^T = \left[\begin{array}\ a_1& a_2& \cdots & a_n \end{array}\right] \left[\begin{array}\ b_1^T\\ b_2^T\\ \vdots \\ b_n^T \end{array}\right] $$

$$ A^TB^T=\left[\begin{array}\ a_1b_1^T + a_2b_2^T+ \cdots + a_nb_n^T \end{array}\right] \qquad(2) $$

EDIT

So the term $\sum_{i=1}^nv_iv_i^T$ is the term in (2). So now if we prove that for $B^TA^T = I$, left inverse is same as right inverse, i.e. $A^TB^T = I$, our work is done. For this proof, check out If $AB = I$ then $BA = I$

Now I know that this is not what you asked. You wanted to understand it in terms of orthogonality of the eigenvectors. For this I have tried to summate $\sum_{i=1}^nv_iv_i^T$ for different matrices and found that for orthonormal columns, the rows are also orthonormal. I have checked so many sources for its proof, but every proof is just utilising the $Q^TQ = Q Q^T = I$, which is what you wanted the proof of in the first place. Arithmetically everything is somehow falling into correct places to make it identity but I'm just unable to formulate it even after I tried with many examples.

So I think my above explanation is arithmetically correct enough for the proof, but in order to utilise the orthonormality of the eigenvectors (i.e. column space of $V$), I have imagined it in the following way:

We know that $v_iv_i^T$ is a projection matrix which projects on to the column space of this projection matrix, which in turn is essentially along the column vector itself since it is a rank-1 matrix.

Now $v_1v_1^T$ will project an n-dimensional vector $x$ along $v_1$, similarly $v_2v_2^T$ along $v_2$ and so on. So when you add all these n projections, you end up with the original vector $x$, thus in a way, sum of all these projection matrices, project the vector onto itself, so it must be identity matrix.

EDIT2:

Extending this visualization of projections to equations based on link provided by Amrit in the comments:

So the n-dimensional vector $x$ can be written as sum of all the projections, with each projection length as $c_i$, we get:

$$ x = c_1v_1+c_2v_2+\cdots+c_nv_n $$

To find each $c_i$, we dot product $x$ with $v_i$ $$ x\cdot v_i = (c_1v_1)\cdot v_i+\cdots + (c_iv_i)\cdot v_i + \cdots + (c_nv_n)\cdot v_i $$ $$ x\cdot v_i = 0+\cdots +c_i+\cdots +0 $$

So rewriting $x$ in terms of these calculated coefficients,

$$ x = (x\cdot v_1)v_1+\cdots +(x\cdot v_n)v_n $$ $$ x = \sum_i(v_i^Tx)v_i $$ $$ \Bigl(\sum_iv_iv_i^T-I_n\Bigr)x=0~\forall x \implies \sum_i v_iv_i^T=I_n $$

Answer 2

Matrix product, usually defined as matrix of inner product of column vectors, coincide with outer sum of outer product (matrices) of column vectors. So you said

I'm struggling to see why this implies that the sum of their outer-products also has to equal the Identity matrix.

The sum of outer-products(of orthonormal basis) equals to matrix of inner products of orthonormal basis. Matrix of inner products of orthonormal basis, is the identity matrix.

Is this answer satisfies you?

Answer 3

While this answer won't give you any real insight, it does prove the fact. Assume $V^\top V= I$. Since we are dealing with square matrices, we have $VV^\top V = VI=V=IV$. But then we find that $VV^\top=I$ by comparing the left and right side of the previous equality. In other words, we get this fact for free.