Let $A,B$ be two $n \times n $ matrices with real entries and $\alpha \in \mathbb{R}, \alpha \neq 0$. It is given that $$A^2-B^2=\alpha(AB-BA)$$ Prove that $(AB-BA)^n=0$.
At first, I tried to prove that $A(AB-BA)=(AB-BA)A$, as it would imply that $(AB-BA)$ is nilpotent and the problem would be solved. But this didn't work.
Then, after writing $\det(A-B)(A+B)=\det(A+B)(A-B)$ and using the given relation I obtained $\det (AB-BA)=0$. But I got stuck and I tried to make a substitution: \begin{align*} X=\frac{1}{2}(A+B) \\ Y=\frac{1}{2}(A-B) \end{align*} The given relation then becomes $$(\alpha+1)XY=(\alpha-1)YX \quad (1)$$ so $XY$ and $YX$ commute. Since $(AB-BA)^n=2^n(YX-XY)^n$, we can expand the latter using the binomial theorem. Combined with $(1)$, I obtained $$(AB-BA)^n=\frac{4^n}{(\alpha-1)^n}(XY)^n=\frac{2^n(-2^n)}{(\alpha+1)^n}(YX)^n$$ and I don't see how to finish this.
