Singular Value Decomposition: Prove that singular values of A are square roots of eigenvalues of both $AA^{T}$ and $A^{T}A$.

Question

From Singular Value Decomposition, we know that:

Any $m$ x $n$ matrix A can be factored into $A=U\Sigma V^{T}$ , where $U$ and $V$ are orthogonal, and $\Sigma$ is of the same size as $A$ with all entries zero except down the main diagonal where the successive entries are $\sigma _{1}\geq ...\geq \sigma _{k} > 0 $ for some $k$ with $k\leq$ min$(m,n)$.

To find $U$, $\Sigma$, and $V$ , we can consider $AA^{T}$ and $A^{T}A$ which are symmetric matrices.

$AA^{T}=(U\Sigma V^{T})(V\Sigma^{T} U^{T})=U(\Sigma \Sigma^{T}) U^{T}$ ($\because$ V is orthogonal implies $V^{T}V=I_{n}$) $A^{T}A=(V\Sigma^{T} U^{T})(U\Sigma V^{T})=V(\Sigma^{T} \Sigma)V^{T} $ ($\because$ U is orthogonal implies $U^{T}U=I_{m}$)

From Spectral Theorem, I understand that, since $AA^{T}$ and $A^{T}A$ are symmetric matrices, $U$ must be the eigenvector matrix for $AA^{T}$, and $\Sigma \Sigma^{T}$ is the eigenvalue matrix for $AA^{T}$; whereas $V$ must be the eigenvector matrix for $A^{T}A$, and $\Sigma^{T} \Sigma$ is the eigenvalue matrix for $A^{T}A$.

But, I don't understand why the $k$ singular values on the diagonal of $\Sigma$ are the square roots of the nonzero eigenvalues of both $AA^{T}$ and $A^{T}A$. It seems like this is only true if $\Sigma \Sigma^{T}$=$\Sigma ^{2}$ and $\Sigma^{T} \Sigma$=$\Sigma ^{2}$ . But $\Sigma \Sigma^{T}$ is $m$ x $m$ matrix, whereas $\Sigma^{T} \Sigma$ is $n$ x $n$ matrix. How can both of them be equal to $\Sigma^{2}$ ?

I'm so confused. :(

Answer 1

Let's just multiply some matrices. \begin{align*} \begin{pmatrix} 2 & 0 & 0 \\ 0 & 3 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix} \begin{pmatrix} 2 & 0 & 0 & 0 \\ 0 & 3 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{pmatrix} &= \begin{pmatrix} 4 & 0 & 0 & 0 \\ 0 & 9 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{pmatrix}, \\ \begin{pmatrix} 2 & 0 & 0 & 0 \\ 0 & 3 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{pmatrix} \begin{pmatrix} 2 & 0 & 0 \\ 0 & 3 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix} &= \begin{pmatrix} 4 & 0 & 0 \\ 0 & 9 & 0 \\ 0 & 0 & 0 \end{pmatrix}. \end{align*} See how the non-zero diagonal entries of $\Sigma\Sigma^T$ and $\Sigma^T\Sigma$ agree?

Answer 2

The $\Sigma$ matrix is a sabot matrix which insures conformability between $\mathbf{U}$ and $\mathbf{V}^{*}$. In block form, use the diagonal matrix of singular values $\mathbf{S}_{\rho \times \rho}$ where $\rho$ is the matrix rank: $$ \Sigma = \left( \begin{array}{cc} \mathbf{S} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \\ \end{array} \right)_{m\times n}, \quad % \Sigma^{\mathrm{T}} = \left( \begin{array}{cc} \mathbf{S} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \\ \end{array} \right)_{n\times m} % \Sigma^{\dagger} = \left( \begin{array}{cc} \mathbf{S}^{-1} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \\ \end{array} \right)_{n\times m} $$

For example, if the target matrix $\mathbf{A}$ has full column rank $$ \Sigma = \left( \begin{array}{c} \mathbf{S} \\ \mathbf{0} \end{array} \right)_{m\times n}, \quad % \Sigma^{\mathrm{T}} = \left( \begin{array}{cc} \mathbf{S} & \mathbf{0} \\ \end{array} \right)_{n\times m} % \Sigma^{\dagger} = \left( \begin{array}{cc} \mathbf{S}^{-1} & \mathbf{0} \\ \end{array} \right)_{n\times m} $$ and $$\Sigma^{\mathrm{T}}\Sigma = \mathbf{S}^{2}.$$

For an example showing $\Sigma$ gymnastics, see SVD and linear least squares problem.