Proof that border rank and the rank of a matrix (order 2 tensors) are equivalent

Question

Recall the definition of border rank for a matrix (order 2 tensor, which can be easily be extended to any order tensor):

border-rank(T) is the minimum r such that $\forall \epsilon > 0$ there exists an approximation $T'=\sum^r_{i=1} u_i \otimes v_i , s.t. \|T - \sum^r_{i=1} u_i \otimes v_i \| \leq \epsilon $

The norm we use is not important (I was told).

Intuitively, what this definition (kind of) means is:

What is the smallest r, so that when I restrict myself to rank r tensors, I can get arbitrarily close approximations to T.

Also

You will always need at least rank r decomposition to be able to arbitrarily approximate a tensor T.

I was told that the rank of a matrix and the border-rank of a matrix are the the same. I wanted to see a rigorous proof for this but was unsure how to go about it.

The intuition/ideas that I do have is that for any matrix there exists the SVD for it. i.e. $M = \sum^{r}_{i=1} \sigma_i u_i v^T_i$ and there is an intrinsic limit to how well one can approximate M. For me it seems that it mostly depends on the truncated SVD. So it seems that if we choose a sufficiently small $\epsilon$, then for order 2 tensors (matrices), we cannot approximate it well enough.

Is my intuition wrong or how would a proof of rank and border-rank equivalence go?

For completeness this is were I read this claim of the equivalence:

Link to the whole document (page 27):

http://people.csail.mit.edu/moitra/docs/bookex.pdf

---

As a side note, the notes referenced, give a link to show that there is a difference between border rank and rank for tensors of order 3.

Answer 1

Here is the proof that for matrices, rank and border rank are equal:

Let $T$ be an $m \times n$ matrix of border rank $r$ and rank $R$. We have $r \leq R$. Let $T_k \to T$ be a sequence of matrices which approach $T$ (say, entry-wise) and with each $T_k$ having rank less than or equal to $r$. (Well, after finitely many, they will need to have rank equal to $r$, but it doesn't matter.)

Now, since each $T_k$ has rank less than or equal to $r$, then every size $r+1$ minor of $T_k$ is equal to zero. (By "minor" I mean determinant of a submatrix.) By continuity, since determinants are polynomials and hence continuous, we see that therefore each size $r+1$ minor of $T$ also vanishes.

And now here is a fact that is true for matrices (order $2$ tensors), but not for higher order tensors: The converse is also true, if every $r+1$ minor of $T$ vanishes then $T$ has rank less than or equal to $r$. This proves that $R \leq r$.

Regarding the fact that the norm "doesn't matter": this is the fact that in a finite-dimensional real vector space, any two norms generate the same topology (e.g., Understanding of the theorem that all norms are equivalent in finite dimensional vector spaces). So a sequence $T_k \to T$ with respect to your first norm, meaning $\|T_k-T\|_1 \to 0$, if and only if $T_k \to T$ with respect to the second norm, $\|T_k-T\|_2 \to 0$. In this sense, to say whether a sequence converges to a certain limit, it doesn't matter what norm you use.

For other purposes such as applications or computation perhaps it might make a difference. If you want to find an approximation within some $\epsilon$, then you have to know what norm (or metric) you are using. And different norms have different computational properties.

Finally, regarding SVD and intrinsic limits to approximation: Truncated SVD is one way to get an approximation. It's the "closest" low-rank approximation. But note that these approximations definitely have strictly lower rank than the matrix you're approximating. So for example you may have a matrix $T$ of rank $5$ and truncate to get a matrix of rank $4$; there will not be any closer rank-$4$ matrix. That lines up with $T$ not having border rank $4$. Indeed, $T$ has border rank $5$, because a matrix's border rank is equal to its rank. There is indeed some $\epsilon>0$ so that no matrix within $\epsilon$ of $T$ (with respect to some norm) has rank $4$, or $3$, or anything less than $5$. This could be a proof of the statement that border rank and rank are equal, but first you would have to prove that the truncated SVD is the closest low-rank approximation, and I suspect it would get a bit circular.

It fails for higher-order tensors. An order-$3$ tensor $T$ ("three-dimensional array of numbers") could perfectly well have rank $5$ and border rank $4$, meaning that for any $\epsilon>0$ there are rank $4$ tensors within $\epsilon$ of $T$. It can happen that there is no "closest" rank $4$ approximation.