5

We know from linear algebra, the least square solution of linear equation system : $$Ax=b$$ always exists. That is, the equation $$A^TAx=A^Tb$$ always has at least one solution. Most of us explain it existence by interpreting least square problem as a matter of orthogonal projection. It's correct, I know. But can we prove this by showing that the rank of matrix $$A^TA$$ is equal to the rank of the augmented matrix:$$[A^TA,A^Tb]$$for any given vector b?

I've worked for hours but failed to show that.

Although there is a similar thread in mathematics.SX, but I personally think there is no good answer presented there. So I address this problem again.

Thanks at first!

[update]

Now I come up with a proof for this proposition:

suppose that rank $A^TA$=k, and the matrix $[A^TA,A^Tb]$ has one more column than $[A^TA]$, so rank $[A^TA,A^Tb]\ge k$. But on the other hand, we have $[A^TA,A^Tb]=A^T[A,b]$, using the rank inequality, we have $rank [A^TA,A^Tb]\le rank A^T$. Since (we can prove that) $ rank A^TA= rank A= rank A^T$ for any given matrix A. So the inequality is actually $rank [A^TA,A^Tb]\le k$. Combining the two inequalities, we have $rank [A^TA,A^Tb]=rank A^TA$. Q.E.D.

Robert Fan
  • 759
  • All you need is that $A^T A$ is square to guarantee at least one solution... And $A^T A$ is square for any $A$. – AlexR Oct 17 '13 at 12:48
  • @AlexR:But $A^TA$ can be not inversable – Robert Fan Oct 17 '13 at 12:52
  • Note "at least one" solution. Else you'd need the projection argument in some form (The image of $A^T$ in the $A^T b$ does this) – AlexR Oct 17 '13 at 12:52
  • @AlexR: Why the case that the rank of $A^TA$ is k-1, while the rank of $[A^TA,A^Tb]$ is k doesn't exit? can you figure it out? – Robert Fan Oct 17 '13 at 12:56
  • 1
    Actually all you need is that the right hand side vector of $A^TAx=A^Tb$ lies in the column space of the matrix $A^TA$ and hence this system has always at least one solution, which is not necessarily unique unless $A$ has full column rank. Obviously the fact that the matrix is square does not mean anything with respect to the fact whether this system is solvable or not. – Algebraic Pavel Oct 17 '13 at 13:04
  • 2
    @AlexR I don't think the "squareness" of the matrix implies anything about solvability of the system. – Algebraic Pavel Oct 17 '13 at 13:09
  • If your solution is significantly different from the ones at the other question, I'd encourage you to post it as an answer to the other question, so everything is in one place the next time someone comes looking. – Gerry Myerson Oct 20 '13 at 00:42
  • @AlgebraicPavel can you add why $A^T b$ is necessarily in the column-space of $A^T A$? – Klint Qinami Aug 11 '16 at 20:06

1 Answers1

3

You can decompose $b$ into a vector $Av$ in the column space of $A$ and a vector $w$ orthogonal to that column space: $Av+w=b$. Then $A^tAv+A^tw=A^tb$. But $w$ is orthogonal to the row space of $A^t$, so $A^tw=0$, so $A^tAv=A^tb$.

Gerry Myerson
  • 179,216