Let $A$ and $B$ be complex $n\times n$ matrices such that $AB−BA$ is invertible and such that $A^2+B^2=c(AB−BA)$ for some rational number $c$. Prove $c\in\{−1,0,1\}$ and show that $n$ is a multiple of $4$ when $c\neq0$.
Please give some hints for this problem. I could not find a way to proceed. I had taken determinant on both sides. But it does not helping much
This is user91684's repertoire. Since he/she has left this community, I will outline a possible approach using one of his/her favourite tricks (which for instance was applied in this answer to a similar question).
Let $A=X+iY$ and $B=iX+Y$ or equivalently, let $X=\frac12(A-iB)$ and $Y=\frac{1}{2i}(A+iB)$. The equality $A^2+B^2=c(AB-BA)$ can then be rewritten as $$ (c-i)XY=(c+i)YX. $$ Since $AB-BA=2(XY-YX)$ is invertible, neither $XY$ nor $YX$ is zero. Therefore $c\ne\pm i$. It follows that $XY=\omega YX$ where $\omega=\frac{c+i}{c-i}\ne 1$. Hence $AB-BA=2(XY-YX)=2(\omega-1)YX$ and $YX$ is invertible. Since $XY$ and $YX$ have identical spectra, the equality $XY=\omega YX$ implies that the spectrum of $XY$ is invariant when it is multiplied by $\omega$. It follows that $\omega$ is some primitive $m$-th root of unity and the spectrum of $XY$ is then a disjoint union of $m$-cycles of the form $\lambda\{1,\omega,\omega^2,\ldots,\omega^{m-1}\}$. Thus $m$ divides $n$. If you can show that the only values of $c$ such that $\omega=\frac{c+i}{c-i}$ is a primitive root of unity are $0,1$ and $-1$, then you are done because $m=4$ (and $\omega=\pm i$) when $c=\pm1$.
Remark. Two matrices $X$ and $Y$ that satisfies the equation $XY=\omega YX$ are said to be quasi-commutative or $\omega$-commutative. According to the aforementioned answer by user91684, there is an interesting result, discovered by Drazin, about quasi-commutativity. See section 5 of Holtz, Mehrmann and Schneider, Potter, Wielandt, and Drazin on the Matrix Equation $AB=\omega BA$: New Answers to Old Questions, Amer. Math. Monthly, 111:8 (2004), 655-667.
This can't be right. There are $n^2$ equations and $2 n^2$ unknown over complex numbers - there will be solution almost always.
