Let $A, B \in M_n(\mathbb{C})$ be two $n \times n$ matrices such that
$$A^2B+BA^2=2ABA$$
Prove that there exists a positive integer $k$ such that $(AB-BA)^k=0$.
Here is the source of the problem. Under the comment by Carcul, I don't understand why $[A,[A,B]]=0$ implies that $[A,B]$ is a nilpotent matrix.
Can anyone explain it to me?
Clarification: I have problem understanding the solution given by Carcul. I don't see the link between $[A,[A,B]]=0$ and $[A,B]$ being nilpotent.
From $[A,[A,B]] = 0$, we decuce by induction that
$$ [A^k, B] = kA^{k-1}[A,B] \tag 1 $$
For $k = 0$, (1) is obviuos, if (1) holds for $k-1$, we have \begin{align*} [A^k, B] &= A^kB - BA^k\\ &= A^{k-1}AB - BAA^{k-1}\\ &= A^{k-1}[A,B] + A^{k-1}BA - BA^{k-1}A\\ &= A^{k-1}[A,B] + [A^{k-1},B]A\\ &= A^{k-1}[A,B] + (k-1)A^{k-2}[A,B]A\\ &= kA^{k-1}[A,B] \qquad \text{ as $[A,B]A = A[A,B]$} \end{align*} As $[\cdot, B]$, is linear, we have that for any polynomial $p$ we have $$ [p(A), B] = p'(A)[A,B] $$ Now let $\mu_A$ denote the minimal polynomial of $A$, we will show by induction that
$$ \tag 2 \mu^{(k)}_A(A)[A,B]^{2^k- 1} = 0 $$
holds for any $k$. For $k = 0$, we have nothing to show, as $\mu_A(A) = 0$ by definition of the minimal polynomial. Suppose $k \ge 1$ and (2) holds for $k-1$, then \begin{align*} 0 &= [\mu^{(k-1)}_A(A)[A,B]^{2^{k-1}-1}, B]\\ &= [\mu^{(k-1)}_A(A), B][A,B]^{2^{k-1}-1} + \mu_A^{(k-1)}(A)[[A,B]^{2^k-1}, B]\\ &= \mu^{(k)}_A(A)[A,B]^{2^{k-1}} + \mu_A^{(k-1)}(A)\sum_{l=1}^{2^{k-1}-1} [A,B]^{l-1} [[A,B],B][A,B]^{2^{k-1}-l}\\ \end{align*} As $[A,[A,B]] = 0$, any polynomial in $A$ commutes with $[A,B]$. Multiplying the last equation with $[A,B]^{2^{k-1}-1}$ from the left, we have \begin{align*} 0 &= [A,B]^{2^{k-1}-1}\mu^{(k)}_A(A)[A,B]^{2^{k-1}} + [A,B]^{2^{k-1}-1}\mu_A^{(k-1)}(A)\sum_{l=1}^{2^{k-1}-1} [A,B]^{l-1} [[A,B],B][A,B]^{2^{k-1}-l}\\ &= \mu^{(k)}_A(A)[A,B]^{2^k-1} + \sum_{l=1}^{2^{k-1}-1}\mu_A^{(k-1)}(A) [A,B]^{2^k-1 + l-1} [[A,B],B][A,B]^{2^{k-1}-l}\\ &= \mu^{(k)}_A(A)[A,B]^{2^k-1} \quad\text{by induction hypothesis} \end{align*} This proves (2).
Now, in (2), let $k = \deg \mu_A$ to get $$ \tag 3 (\deg \mu_A)![A,B]^{2^{\deg \mu_A} - 1} = 0 $$
Hence, $[A,B]$ is nilpotent.
Due to the assumptions $$ (AB-BA)^{k+1} = (AB-BA)(AB-BA)^k = AB(AB-BA)^k - B(AB-BA)^kA. $$ Then we find for the trace $$ tr( (AB-BA)^{k+1} ) = tr \left( A \big(B(AB-BA)^k\big) - \big(B(AB-BA)^k\big)A\right)=0, $$ since $tr(CD)=tr(DC)$. This shows that $$ tr((AB-BA)^{k})=0 $$ for all $k\ge 1$. This implies that $AB-BA$ is nilpotent:
Traces of all positive powers of a matrix are zero implies it is nilpotent
Let $X = AB - BA$
From the assumptions $A^2B - ABA = ABA - AB^2 \iff A(AB-BA) = (AB-BA)A \iff AX = XA $
By induction: $$ AX^k = X^kA \text{ } \forall k \in \mathbb{N} $$
$X^k = X^{k-1}X = X^{k-1}(AB-BA) = X^{k-1}AB - X^{k-1}BA = AX^{k-1}B - X^{k-1}BA \Rightarrow $
$\operatorname{tr}(X^k) = \operatorname{tr}(AX^{k-1}B)- \operatorname{tr}(X^{k-1}BA) = 0 \text{ } \forall k \in \{1,2,\dots,n\}$
One can show that this leads to the nilpotence of $X = AB-BA$.
Why does $\operatorname{tr}(X^k) \text{ } \forall k \in \{1,2,...,n\}$ imply the nilpotence of $X$ ?
Let $\lambda_1, \lambda_2, \dots \lambda_n $ be the eigenvalues of $X$. Then $\operatorname{tr}(X^k) = \sum\limits_{i=1}^n \lambda_i^k $. The coefficients of the characteristic polynomial ( besides the coefficient of $x^n$ ) are the symmetric elementary polynomial in variables $\lambda_1, \lambda_2, \dots \lambda_n$. Newton's Identities say that: $$ ke_k = \sum_{i=1}^k(-1)^{i-1} e_{k - i} p_i, $$ where $e_k$ are the symmetric polynomials and $p_k = \operatorname{tr}(X^k).$
Because $\operatorname{tr}(X^k) = 0$ we can conclude that $e_1 = e_2 = \dots = e_n = 0$ so the characteristic polynomial of $X$ is $X^n$. By Cayley Hamilton $X^n = 0$.
