4

I'm studying Principle Component Analysis (PCA), and came across this post. In which it's written that diagonalization of co-variance matrix ($C$) can be given by $$C = VLV^T$$

But as per difference between SVD and Diagonalization and this post , it's clear that diagonalization of any matrix can be given by: $$C = VLV^{-1}$$

So why the definition of SVD and diagonalization is same here?

Parcly Taxel
  • 103,344
Kaushal28
  • 685
  • Is the matrix that you're diagonalizing / decomposing symmetric? –  Jul 21 '18 at 05:13
  • Yes, Co-variance matrix is always symmetric. – Kaushal28 Jul 21 '18 at 05:15
  • Then your singular values are exactly the eigenvalues - try to verify this on your own. –  Jul 21 '18 at 05:17
  • I can tell you that a diagonalization of any matrix can be given by the formula you give with $V$ and $V^-1$. As a corolary of the spectral theorem for autoadjoints lineal transformations it is proved that any symmetric matrix is diagonalizable and what is more $V$ can be obtained such that $V^{-1} = V^t$. What I can not find is where is the definition of diagonalization and SVD in the link you post (the last here). – HFKy Jul 21 '18 at 05:22
  • your singular values are the square root of the eigenvalues –  Jul 21 '18 at 05:25
  • A sum of self-outer products is always symmetric matrix which is always positive definite with an ON eigensystem. – mathreadler Jul 21 '18 at 05:35
  • A symmetric matrix always has an orthonormal basis of eigenvectors, so $V$ is orthogonal and there is no difference between $C = VLV^{-1}$ and $C = VLV^T$. –  Jul 21 '18 at 06:06
  • Sorry but I don't know the meaning of orthonormal basis. @Rahul – Kaushal28 Jul 21 '18 at 07:43
  • A symmetric matrix always has a diagonalization in which $V$ is orthogonal, so there is no difference between $C=VLV^{-1}$ and $C=VLV^T$. –  Jul 21 '18 at 09:59

1 Answers1

3

The SVD is a generalization of the eigendecomposition. The SVD is the following.

Suppose $ A \in \mathbb{C}^{m \times n}$

now

$$A = U \Sigma V^{T} $$

where $U,V^{T}$ are orthogonal matrices and $\Sigma $ is a diagonal matrix of singular values. The connection comes here when forming the covariance matrix

$$AA^{T} = (U \Sigma V^{T}) (U \Sigma V^{T})^{T} $$ $$AA^{T} = (U \Sigma V^{T}) (V \Sigma^{T}U^{T})$$ $$ AA^{T} = U \Sigma V^{T} V \Sigma^{T} U^{T}$$

Now $VV^{T} =V^{T}V = I $ $$ AA^{T} = U \Sigma \Sigma^{T} U^{T} $$ Also $ \Sigma^{T} = \Sigma $ $$ AA^{T} = U\Sigma^{2} U^{T}$$ Now we have $ \Sigma^{2} = \Lambda $ $$ AA^{T} = U \Lambda U^{T}$$

The actual way you compute the SVD is pretty similar to the eigendecomp.

In respect to the PCA, it is telling you specifically in the answer you have take the covariance matrix and normalize it (centering). Then it only take the left singular vectors and singular values I believe while truncating it.

A truncated SVD is like this.

$$A_{k} = U_{k}\Sigma_{k} V_{k}^{T} $$

this means the following $$A_{k} = \sum_{i=1}^{k} \sigma_{i} u_{i} v_{i}^{T} $$

So you actually read that they aren't the same. It uses the SVD in forming because it is simpler. The last part states

The product $$ U_{k}\Sigma_{k}$$

gives us a reduction in the dimensionality which contains the first k principal components here we then multiply be the principal axis

$$X_{k} = U_{k}\Sigma_{k} V_{k}^{T} $$

This is commonly referred to as a truncated SVD.