Problems of this sort can be solved by the technique illustrated in a previous answer by user91684. The idea is to pick four coefficients $p,q,r,s$ appropriately and put $A=pU+qV$ and $B=rU+sV$, such that $A^2+xB^2-y(AB-BA)$ is in the form of
$$
c(VU-\omega UV)
$$
where $c\ne0$ and $\omega$ is neither zero nor a root of unity. If this can be done, then the condition $A^2+xB^2=y(AB-BA)$ implies that $VU=\omega UV$. However, as $UV$ and $VU$ always have the same spectrum, the equality $VU=\omega UV$ implies that $UV$ must be nilpotent. Therefore $AB-BA$ is nilpotent, because it is a linear combination of $UV$ and $VU$, and in turn a scalar multiple of $UV$.
To be concrete, let $A=i\sqrt{x}(U-V)$ and $B=U+V$. Then
$$
A^2+xB^2=-x(U^2-UV-VU+V^2)+x(U^2+UV+VU+V^2)=2x(UV+VU)
$$
and
$$
y(AB-BA)=2iy\sqrt{x}(UV-VU).\tag{1}
$$
The condition $A^2+xB^2=y(AB-BA)$ is therefore equivalent to $2x(UV+VU)=2iy\sqrt{x}(UV-VU)$, or $-i\sqrt{x}(UV+VU)=y(UV-VU)$, or
$$
VU=\omega UV,
$$
where $\omega=\frac{y+i\sqrt{x}}{y-i\sqrt{x}}$.
Note that $\operatorname{Re}\omega
=\operatorname{Re}\left(\frac{(y+i\sqrt{x})^2}{y^2+x}\right)
=\frac{y^2-x}{y^2+x}$. So, $\frac{1}{\pi}\arg\omega=\frac{1}{\pi}\arccos\left(\frac{y^2-x}{y^2+x}\right)$ is not a rational number, meaning that $\omega$ is not a root of unity. Hence the equality $VU=\omega UV$ implies that $VU$ and $UV$ are nilpotent, and from $(1)$ we see that $AB-BA=2i\sqrt{x}(UV-VU)=2i\sqrt{x}(1-\omega)UV$ is also nilpotent.
One can actually obtain a stronger conclusion. As $UV=\omega VU$ where $\omega$ is not a primitive root of unity, by a result of Drazin (see theorems 6 and 7 in Holtz et al., Potter, Wielandt, and Drain on the Matrix Equation $AB=\omega BA$: New Answers to Old Questions), $U$ and $V$ must be simultaneously triangulable. Hence so do $A$ and $B$. This explicitly explains why $AB-BA$ is nilpotent.
