1

I'm working on a least-squares problem with an overdetermined matrix, and I've noticed that changing the data very slightly leads to a huge change in my solution (in this case, plotting an elliptical path).

I suspect that this is because the changes in data cause the matrix to become very close to being rank-deficient, but I'm not sure how to confirm this. Is there a way to measure whether a matrix is "close" to being rank-deficient?

Edit: I've been thinking about how if they were square, perhaps the determinant of the altered matrix would be closer to zero than the original. Perhaps I could compute the determinants of each matrix multiplied by its transpose to achieve a similar conclusion? Or this just wishful thinking?

NNN
  • 1,852

4 Answers4

3

I will extend what @Chris Hansen said. A good way to approach this problem is using the Singular Value Decomposition of the matrix $A = U \Sigma V^*$. When you compute it, you want to look at the magnitude of the smallest few singular values and (potentially) make them zero. You will obtain a new decomposition $A'= U \Sigma'V^*$ which is the best approximation of A of the corresponding rank and try to solve the problem for $A'$ instead of $A$. Look here for more information how to solve a least squares problem via SVD.

By the way, the instability that you mention might arise because you are solving via the Normal equations. If this is the case: don't use them because they are extremely unstable. Use SVD or QR decomposition.

Community
  • 1
Milen Ivanov
  • 1,424
2

"A matrix gets the rank it deserves." The rank of a matrix is the size of any largest nonsingular square submatrix (permuting rows and columns). A square matrix is singular when its determinant is "zero." But what is zero in finite precision arithmetic?

My idea is to apply Gaussian Elimination with Complete Pivoting (GECP), where at each step the absolutely largest entry in the residual Schur Complement matrix is selected as the next pivotal element. Rank increases by one at each factorization step until the infinity norm of the remaining residual matrix is less than some specified tolerance, indicating that the remaining residual matrix contains all "zeros." If the residual matrix is null, the matrix has full row and/or column rank.

If GECP has determined that the matrix has rank $r$, I would divide the absolutely smallest pivot by the absolutely largest pivot to get an idea of how close the matrix is to rank $r-1$. If this number is close to the above specified tolerance, the matrix is close to rank $r-1$. If it is significantly larger than the specified tolerance, the matrix is not close to rank $r-1$.

Mason
  • 3,792
Roger
  • 31
  • 1
  • Sorry. I was reviewing this thinking it was a first question and not a first answer. That's my bad. +1 for being a newbie and putting down an actual attempt to answer the question intelligently. – Mason Aug 03 '18 at 02:09
1

The condition number (ratio of largest to smallest singular value) is helpful. See: https://en.wikipedia.org/wiki/Condition_number.

The relationship between the condition number and the resulting sensitivity of solutions to linear equations is discussed in Fundamentals of Matrix Computations (Watkins) and Matrix Computations (Golub and Van Loan). In practice you would like it to be very small compared to the number of digits of precision in your computation. Say 10^3 in most cases with typical floating point.

Chris Hansen
  • 126
  • Ok follow-up question. How do I know whether the condition number is "big enough" to confirm my suspicion? – NNN Mar 10 '16 at 20:47
  • I updated my answer above. The other answer is also helpful as a technique for managing large condition numbers. – Chris Hansen Mar 10 '16 at 21:07
1

Given $\mathbf{A}\in\mathbb{C}^{m\times n}_{\rho}$, and it's ordered singular values $$ \sigma_{1} \ge \sigma_{2} \ge \dots \ge \sigma_{\rho} > 0 $$ for each $k$ < $\rho$, the distance from $\mathbf{A}$ to a matrix of rank $k$ is $$ \sigma_{k+1} = \min_{\text{rank }\mathbf{B} = k} \lVert \mathbf{A} - \mathbf{B} \rVert_{2}. $$

$\S$ 5.12, Carl Meyer, http://bookstore.siam.org/ot71/Matrix Analysis and Applied Linear Algebra

dantopa
  • 10,342