Linear least square solution

Question

The Linear least square solution is obtained by solving $XB =y$ and then $B$ is calculated by following equation

$$B =(X^TX)^{-1}Xy $$

Why we go for a pseudo inverse instead of taking a inverse

$$B = X^{-1}y$$

BTW I think that should be $B=(X^TX)^{-1}X^T{\bf y}$. – David Apr 01 '14 at 05:21 — David, Apr 01 '14 at 05:21

Erik M · Accepted Answer · 2014-04-01T05:27:39.673

2

In least squares problems we are dealing with an overdetermined system. This means that we have more equations than unknowns, thus the matrix $X$ in your problem is not square. The inverse of a matrix is not defined for non-square matrices. One way of getting around this is to take the inverse of the square matrix product $X^TX$ and form the pseudo-inverse $(X^TX)^{-1}X^T$.

edited Apr 01 '14 at 05:27

answered Apr 01 '14 at 05:06

Erik M

2,391

score 0 · Answer 2 · answered Apr 27 '17 at 03:15

If the matrix has full column rank, then the pseudoinverse is the same as the classic inverse. Details are provided in What forms does the Moore-Penrose inverse take under systems with full rank, full column rank, and full row rank? or generalized inverse of a matrix and convergence for singular matrix.

If the matrix has a column rank defect, then the classic inverse does not exist.

In summary, the answer is always expressed in terms of the pseudoinverse. When the classic inverse exists, it is the same as the pseudoinverse.

Linear least square solution

2 Answers2