2

What scenarios would cause $X^TX$ to not be invertible?

Kevin Sai
  • 69
  • The answers below are correct and I'll amplify / clarify that $X^TX$ is invertible if and only if $X$ is of full column rank, in other words the columns are all linearly independent. As others have indicated this is impossible if $n>m$ and that the matrix is full-rank (which for $n>m$ must mean full row rank) is irrelevant. – spaceisdarkgreen Sep 16 '17 at 05:58

3 Answers3

8

Since $n>m$, there is a non-zero vector $v$ such that $Xv=0$, and then $X^TXv=0$ as well. Hence $X^TX$ cannot be invertible.

carmichael561
  • 53,688
  • This follows either from rank-nullity or simply from considering the reduced row-echelon form of $X$. – carmichael561 Sep 16 '17 at 05:45
  • The vectors $Xv$ can only take values in $m$ linearly independent directions (the space is $m$ dimensional). You can feed vectors $v$ into $X$ in $n$ directors, which means that some directions have to map to zero (else, we could get $n$ linearly independent directions out, which is impossible). See also the rank nullity theorem. – Batman Sep 16 '17 at 05:48
  • Rank-nullity in this case states that $\mathrm{rank}(X)+\dim\ker(X)=n$. And since $\mathrm{rank}(X)\leq m<n$, it follows that $\dim\ker(X)$ is positive. – carmichael561 Sep 16 '17 at 05:48
  • If there were no such $v$, then $\ker(X)$ would consist only of the zero vector, and therefore would have dimension zero. – carmichael561 Sep 16 '17 at 06:04
2

You should be able to prove that for any matrices $A,B$ such that $AB$ is defined, rank(AB) $\leq$ min(rank(A),rank(B)) from the definition of rank. See this question and the duplicates therein.

Now, take $A = X^T, B= X$. The rank of $A,B$ is $m$ (since it is full rank). So, $AB$ is a n x n matrix with rank at most $m$ which is less than $n$ so $AB$ is not invertible.

Note that the fact that $X$ is full rank is irrelevant. The only thing we're relying on is $m<n$.

Batman
  • 19,390
0

The answers given are absolutely correct. Let me add my voice to the chorus. This is true if your $X$ is a square matrix.

A Matrix is singular (not invertible) if and only if its determinant is null.

By the properties of the determinant: $$\det{(A)} = \det{(A^T)}$$ And by Binet's theorem: $$\det{(A \cdot B)}=\det{(A)} \det{(B)}$$ Then, you're requesting that: $$\det{(X^T X)} = 0$$ $$\det{(X^T)} \det{(X)} = \det{(X)}^2 = 0$$ $$ \implies \det{(X)} = 0$$ That means you get a singular matrix only by using a singular matrix.

Niki Di Giano
  • 1,708