Exist basis, simultaneously upper-triangular?

Question

Let $A, B \in M_n(\mathbb{C})$ be such that $\text{rank}(AB - BA) \le 1$. Does there exist a basis of $\mathbb{C}^n$ with respect to which $A$ and $B$ are simultaneously upper-triangular?

Answer 1

This is a general truth provided that the underlying field $\mathbb F$ is algebraically closed. For a simple proof, see theorem 9.3.1 (pp.303-304) of Gohberg, Lancaster and Rodman, Invariant Supspaces of Matrices with Applications, SIAM, 2006.

Suppose $A$ is singular and $\operatorname{rank}(AB-BA)=1$. The main idea is that at least one of $\ker(A)$ and $\operatorname{Im}(A)$ is an (nontrivial proper) invariant subspace of $B$. If this can be proved, the rest follows from mathematical induction.

In fact, if $\ker(A)$ is not an invariant subspace of $B$, then $Ax=0\ne ABx$ for some vector $x$. Hence $(AB-BA)x=ABx\ne0$, and $ABx$ spans the column space of the rank-one matrix $AB-BA$. So, for any vector $v$, we have $(AB-BA)v=cABx$ for some scalar $c$. Hence $BAv=ABv-cABx\in\operatorname{Im}(A)$, i.e. $\operatorname{Im}(A)$ is an invariant subspace of $B$.$\ \blacksquare$

---

There are also some related results. E.g. $A,B$ are also triangulable if the field also has characteristic zero, $\operatorname{rank}(AB-BA)\le2$ and a further nilpotency condition is satisfied. See the following papers for some of them:

• Laffey (1980, p.196), Simultaneous triangularization of a pair of matrices whose commutator has rank two, Linear Algebra and Its Applications, 29:195-203 (1980),
• Laffey (1978), Simultaneous triangularization of matrices — low rank cases and the nonderogatory case,
• Robert M. Guralnick (1979), A note on pairs of matrices with rank one commutator.