The problem is from Algebra by Artin, Chapter 1, Miscellaneous Problems, Exercise 8. I have been trying for a long time now to solve it but I am unsuccessful.
Let $A,B$ be $m \times n$ and $n \times m$ matrices. Prove that $I_m - AB$ is invertible if and only if $I_n - BA$ is invertible.
Please provide with only a hint.
Another method:
Let $C= {(I_m - AB)}^{-1}$. The matrix $BCA$ is $n\times n$.
$(I_n - BA)(BCA) = BCA - BABCA $
$= B(C - ABC)A$ $= B[(I_m - AB)C]A$ $= B(I_m)A\ $ (by definition of $C$)
$= BA$
Hence,
$(I_n-BA)(BCA + I_n) = (I_n - BA) (BCA) + (I_n -BA) = BA + (I_n-BA) = I_n$
So, we get that the inverse of $I_n - BA$ is $I_n +BCA$.
There's no real need to actually invoke Sylvester's determinant theorem (although that certainly is much faster).
First show that the (non-zero) eigenvalues of $AB$ and the eigenvalues of $BA$ coincide. If you take the determinant of $I_m - AB$, then you have the characteristic polynomial of $AB$ evaluated at $\lambda = 1$. It follows that the determinant is zero if and only if $1$ is an eigenvalue of $AB$ if and only if $1$ is an eigenvalue of $BA$ if and only if $\det(I_n - BA) = 0$.
Note that a slight adaptation of this argument also provides a proof of Sylvester's determinant theorem different from the one given on Wikipedia.
Hint $\ $ It's a consequence of Sylvester's determinant identity $\rm\:det(1 + AB) = det(1+BA),\:$ which has a very simple universal proof: $ $ over the polynomial ring $\rm\ \mathbb Z[A_{\,i\,j},B_{\,i\,j}\,]\ $ take the determinant of $\rm\, (1+AB)\, A = A\, (1+BA)\ $ then cancel $\rm\, det(A)\ $ (valid since the ring is a domain). $ $ Extend to non-square matrices by padding appropriately with $0$'s and $1$'s to get square matrices. Note that the proof is purely algebraic - it does not require any topological notions (e.g. density).
Alternatively $\ $ Proceed by way of Schur decomposition, namely
$$\rm\left[ \begin{array}{ccc} 1 & \rm A \\ \rm B & 1 \end{array} \right]\ =\ \left[ \begin{array}{ccc} 1 & \rm 0 \\ \rm B & 1 \end{array} \right]\ \left[ \begin{array}{ccc} 1 & \rm 0 \\ \rm 0 & \rm 1-BA \end{array} \right]\ \left[ \begin{array}{ccc} 1 & \rm A \\ \rm 0 & 1 \end{array} \right]$$
$$\rm\phantom{\left[ \begin{array}{ccc} 1 & \rm B \\ \rm A & 1 \end{array} \right]}\ =\ \left[ \begin{array}{ccc} 1 & \rm A \\ \rm 0 & 1 \end{array} \right]\ \left[ \begin{array}{ccc} \rm 1-AB & \rm 0 \\ \rm 0 & \rm 1 \end{array} \right]\ \left[ \begin{array}{ccc} 1 & \rm 0 \\ \rm B & 1 \end{array} \right]$$
See my posts in this sci.math thread on 09 Nov 2007 for further discussion.
We know that a square matrix $M$ is invertible if and only if $$\{\vec x; \vec x M = \vec 0\}=\{\vec 0\}.$$ (The corresponding homogeneous linear system has only trivial solution. The corresponding linear map has trivial kernel.)
So the claim $$I_m-AB\text{ is invertible }\qquad\implies\qquad I_n-BA\text{ is invertible}$$ is equivalent to $$\{\vec x\in F^m; \vec x(I-AB)=\vec 0\}=\{\vec0\} \qquad\implies\qquad \{\vec y\in F^n; \vec y(I-BA)=\vec 0\}=\{\vec0\}.$$ (Where $F$ denotes the base field.)
Proof. So let us assume that $I-AB$ is invertible, i.e., that $\vec x=\vec 0$ is the only vector such that $\vec x(I-AB)=\vec 0$.
Let $\vec y(I-BA)=\vec 0$.
Then we get $$\vec y=\vec yBA\tag{1}$$ and, consequently $\vec y B = \vec y BAB$. This is equivalent to $$\vec yB(I-AB)= \vec 0.\tag{2}$$ This means that $\vec yB=\vec 0$, since $I-AB$ is invertible.
From (1) we now get that $\vec y=\vec yBA=\vec 0A=\vec 0$.
$\square$
