Let $A,B\in M_n(\mathbb{C})$ s.t. $A^3+B^3=0,AB-B^2A^2=I_n$.
According to
one can deduce that $A,B$ are invertible and $BA-A^2B^2=I_n$.
Now, when $n=2$, one has: (*) $A^6=I_2$ and $AB-BA$ is nilpotent.
I prove (*) by a PC computation using the Grobner basis theory.
Question 1. Prove (*) with hand.
Question 2. Does (*) remain true when $n>2$ ?
Here is a proof when $n=2$. The case $n>2$ seems to be much more difficult.
Case 1. $AB=BA$. Alors $A^6=I$ in $M_n(\mathbb{C})$. Indeed $(AB)^2-(AB)+I=0$ implies that $AB$ is similar to $diag(-jI_p,-j^2I_q)$ where $j=\exp(2i\pi/3)$ and $p+q=n$; then $(AB)^3=A^3B^3=-I$ and $A^6=I$.
Case 2. $AB\not= BA$. Necessarily, the system $\{I,A,B\}$ is free. Let $A^2=aI+\alpha A,B^2=bI+\beta B$. Then $AB-BA=B^2A^2-A^2B^2=\alpha\beta(BA-AB)$ and $\alpha\beta=-1$. Moreover, $A^3+B^3=(\alpha a+\beta b)I+(a+\alpha^2)A+(b+\beta^2)B=0$ implies $\alpha a+\beta b=a+\alpha^2=b+\beta^2=0$.
We deduce that $\alpha^6=1,\beta=-1/\alpha$ and finally $A^3=-\alpha^3 I$.
Now we show that $trace((AB-BA)^2)=0$.
$BABA=B^3A^3+BA$ implies $(BA)^2-(BA)+I=0$; thus $BA$ is diagonalizable and $spectrum(BA)=\{-j,-j^2\}$ (because $AB\not= BA$); consequently, $spectrum(AB)=\{-j,-j^2\}$.
Finally $tr((AB-BA)^2)=2tr((AB)^2)-2tr(B^2A^2)=$
$2tr((AB)^2)-2tr(AB-I)=2(j+j^4)-2(-j-1-j^2-1)=0$.
Conclusion. $AB-BA$ is nilpotent and $A,B$ are simultaneously triangularizable.