68

How can I understand that $A^TA$ is invertible if $A$ has independent columns? I found a similar question, phrased the other way around, so I tried to use the theorem

$$ rank(A^TA) \le min(rank(A^T),rank(A)) $$

Given $rank(A) = rank(A^T) = n$ and $A^TA$ produces an $n\times n$ matrix, I can't seem to prove that $rank(A^TA)$ is actually $n$.

I also tried to look at the question another way with the matrices

$$ A^TA = \begin{bmatrix}a_1^T \\ a_2^T \\ \ldots \\ a_n^T \end{bmatrix} \begin{bmatrix}a_1 a_2 \ldots a_n \end{bmatrix} = \begin{bmatrix}A^Ta_1 A^Ta^2 \ldots A^Ta_n\end{bmatrix} $$

But I still can't seem to show that $A^TA$ is invertible. So, how should I get a better understanding of why $A^TA$ is invertible if $A$ has independent columns?

Rodrigo de Azevedo
  • 1
Chewers Jingoist
  • 1,133
  • 3
  • 11
  • 16

5 Answers5

77

Consider the following: $$A^TAx=\mathbf 0$$ Here, $Ax$, an element in the range of $A$, is in the null space of $A^T$. However, the null space of $A^T$ and the range of $A$ are orthogonal complements, so $Ax=\mathbf 0$.

If $A$ has linearly independent columns, then $Ax=\mathbf 0 \implies x=\mathbf 0$, so the null space of $A^TA=\{\mathbf 0\}$. Since $A^TA$ is a square matrix, this means $A^TA$ is invertible.

Noble Mushtak
  • 18,402
  • 28
  • 44
  • This answer uses vocabulary that is much more familiar than the other answer you linked in the comments. Thanks! – Chewers Jingoist Jun 26 '16 at 23:54
  • @ChewersJingoist Glad I could help! – Noble Mushtak Jun 26 '16 at 23:59
  • 2
    I liked @littleO's elegant proof but, going against popular votes, I'm accepting this answer because it gave me a better understanding. – Chewers Jingoist Jun 27 '16 at 00:16
  • How is the proof not even shorter? If we want $A^TAx=0$ only with the zero vector then its obvious thats the case by definition because $A$ has independent columns. End of proof. Why are you conjuring at the beginning of your answer anything about the null space of $A^T$ (known as the left null space)? Isn't it already done pretty much immediately? – Charlie Parker Oct 18 '17 at 21:06
  • 2
    @CharlieParker I am pretty sure your proof is flawed. You do not consider the case where $Ax \neq 0$, but $A^TAx=0$ because $Ax$ is in the null space of $A^T$. In order to show this can't happen, I made an argument using the null space of $A^T$ to show $Ax$ must equal $0$. – Noble Mushtak Oct 18 '17 at 21:54
  • @NobleMushtak yup I noticed it after I thought about it about it a bit more. There could be some $x \neq 0$ s.t. $A^T (Ax) = 0$. My argument only shows that $N(A^TA)$ contains zero. Your proof is right. You earned yourself my up vote. I feel your proof is a lot more insightful, I hate the $| Ax |^2 = 0$ proof. Its random and unintuitive for me, feels it came out of nowhere. Maybe your proof will stick in my brain better, seems more natural. Thanks!!! – Charlie Parker Oct 18 '17 at 21:57
  • @NobleMushtak sorry I thought I got it and then I got confused again. $Ax \in C(A) $ but $N(A^T) \perp C(A)$ for all element of $C(A)$, it doesn't mean its $Ax=0$, no? I guess your second sentence saying that $N(A^T)$ is the orthogonal complement of $C(A)$ just says that $\forall x N(A^T), y \in C(A), x^T y = 0$, I don't quite understand how exactly you manage to conclude $Ax=0$. How did you do that? Obviously thats really important cuz then you can proceed to your second paragraph. – Charlie Parker Oct 18 '17 at 22:41
  • 4
    @CharlieParker Sorry for the late response. $Ax \in N(A^T)$ because we have $A^T(Ax)=0$. Also, $Ax \in C(A)$, as you said. Therefore, $Ax$ is in both $N(A^T)$ and $C(A)$. However, the only vector that can be in both a subspace and the orthogonal complement of its subspace is the zero vector. Thus, $Ax$ must be the zero vector. – Noble Mushtak Nov 05 '17 at 14:50
  • 3
    @CharlieParker Here is an article proving that the intersection of a subspace and its orthogonal complement is just the zero vector. – Noble Mushtak Nov 05 '17 at 14:52
  • I provide a short proof of that $\textit{the intersection of a subspace and its orthogonal complement is just the zero vector}$ here. Let $N$ be the intersection of a subspace $S$ and its orthogonal complement $O$. Suppose $N$ contains non-zero vector(s). Then there exists some non-zero vector $n$ in $N$, which implies $n\cdot n=| n|^2>0$. Note that $n\in S$ and $n\in O$. We shall have $n\cdot n=0$, where we view the first and second $n$ as vectors in $S$ and $O$ respectively. It is a contradiction. On the other hand, obviously $0\in N$. Therefore, we have proved that $N={0}$. – Sam Wong Mar 24 '24 at 13:08
50

If $A $ is a real $m \times n $ matrix then $A $ and $A^T A $ have the same null space. Proof: $A^TA x =0\implies x^T A^T Ax =0 \implies (Ax)^TAx=0 \implies \|Ax\|^2 = 0 \implies Ax = 0 $.

littleO
  • 51,938
  • 1
    I was also wondering why $N(A^TA) = N(A)$ but I didn't think that would be related to my question. Thanks! – Chewers Jingoist Jun 27 '16 at 00:03
  • 1
    Is it true that anything multiplied on the left leaves the null space unaffected? i.e. $N(CA) = N(A)$ for all $C$? – Charlie Parker Oct 18 '17 at 21:09
  • 1
    Also, at what point are you using that the matrix $A$ has column rank $r$? I'm curious because your proof seems a lot less intuitive than what the simpler argument than: If we want $A^TAx=0$ only with the zero vector then its obvious thats the case by definition because $A$ has independent columns, so only $Ax=0$ and thats true too for $A^T A $. – Charlie Parker Oct 18 '17 at 21:11
  • @CharlieParker Let's use 1-by-1 matrices, so say $C=[0]$ and $A=[1]$. I think it is pretty obvious why $CA=[0]$ and $A=[1]$ have different null spaces. – Noble Mushtak Oct 18 '17 at 21:54
  • 4
    @CharlieParker Also, in this answer, littleO only showed that $A$ and $A^TA$ had the same null space. However, the full proof is a bit more intricate than this: Since $A$ has column rank $r$ (i.e. independent columns), it has a trivial null space. Thus, by the above, $A^TA$ also has a trivial null space. Therefore, since $A^TA$ is a square matrix and has a trivial null space, it is invertible. – Noble Mushtak Oct 18 '17 at 21:58
  • @NobleMushtak would an invertible $C$ do the trick? is that the only type of matrix that leave the null space of A $N(A)$ untouched? – Charlie Parker Oct 18 '17 at 22:02
  • 1
    @CharlieParker Yes, an invertible $C$ would not change the null space. However, it is not the only type of matrix that works. As littleO showed, $A^T$ also works, even if $A$ does not have independent columns. – Noble Mushtak Oct 18 '17 at 22:10
7

Let $A \in \mathbb R^{m \times n}$. Note that

$$f (x) := x^T A^T A x = \|A x\|_2^2$$

is positive semidefinite. Function $f$ vanishes when $A x = 0_m$. If $A$ has full column rank, i.e., if its $n$ columns are linearly independent, then $A x =0_m$ implies that $x = 0_n$, i.e., $f$ is positive definite and, hence, $A^T A$ is positive definite and, thus, invertible.

Rodrigo de Azevedo
  • 1
2

I think it need be mentioned that we deal with the real field, as the mentioned both facts may not be true for arbitrary field $F$.

Example 1. Let $F=\mathbb Z_5$ and $A=\left[\begin{smallmatrix} 1 & 3\\ 2 & 1\\ 0 & 1 \end{smallmatrix}\right]$. Then $A$ has two independent columns, but $A^T A=\left[\begin{smallmatrix} 0 & 0\\ 0 & 1 \end{smallmatrix}\right]$ is not invertible.

Example 2. Let $F$ and $A$ be the same as above. Then $A=\left[\begin{smallmatrix} 1 & 3\\ 2 & 1\\ 0 & 1 \end{smallmatrix}\right]\sim \left[\begin{smallmatrix} 1 & 0\\ 0 & 1\\ 0 & 0 \end{smallmatrix}\right]$, i.e., ${\rm null}(A)=\{0\}$. But ${\rm null}(A^TA)={\rm span}(\left[\begin{smallmatrix} 1 \\ 0 \end{smallmatrix}\right])$ not equal to ${\rm null}(A)$.

V. Mikaelian
  • 106
0

Suppose A is a $m \times n$ matrix ($m\geq n$). Since $A$ has linearly independent columns, by QR decomposition $A=QR$ where $Q$ is a $m \times n$ matrix with orthonormal columns and $R$ is a $n \times n$ invertible triangular matrix.

Thus $A^TA=(QR)^T(QR)=R^T(Q^TQ)R=R^TR$. Since $R^T$ and $R$ are both invertible,$A^TA$ is invertible.

lovetl2002
  • 143