Let $A,B$ be two real $n \times n$ matrices. If $C=AB-BA$ is an invertible matrix and $A^2+B^2=\sqrt{3}C$, show that $n$ cannot be a prime number.
Sorry, I've not been able to make progress even after trying this for hours. I'm not really sure how to link the information given with the order of the matrix not being a prime number. It seems as though it has to do something with the $\sqrt{3}$ factor.
Note that if $M$ is a $n \times n$ matrix, and $\overline{M}$ is the matrix $M$ with entries conjugated, then $\det(M\overline{M}) \in \mathbb{R}$. So, consider $M=A+iB$. So, $\overline{M}=A-iB$. Thus, $$M\overline{M}=(A+iB)(A-iB)=A^2+B^2-i(AB-BA)=\sqrt{3}C-iC=(\sqrt{3}-i)C$$ So, $\det(M\overline{M})=(\sqrt{3}-i)^n\det{C} \in \mathbb{R}$. Can you proceed?
Generally, if $aA^2+bB^2=cC$, then $A^2+\frac{b}{a}B^2=\frac{c}{a}C$, then consider $M=A+i\sqrt{\frac{b}{a}}B$ and proceed as before.
The statement is true even if $A$ and $B$ are complex matrices. Let $A=X+iY$ and $B=iX+Y$, i.e. let $X=\frac12(A-iB)$ and $Y=\frac12(B-iA)$. Then the condition $A^2+B^2=\sqrt{3}C$ is equivalent to $$ XY=\omega YX\tag{1} $$ where $\omega=e^{i\pi/3}$ and $$ C=AB-BA=2(XY-YX)=2(\omega-1)YX. $$ Since $C$ is invertible, so is $YX$. Therefore, by taking determinants on both sides of $(1)$, we get $\omega^n=1$. Hence $6|n$ and $n$ is a composite number.
The main trick above is to use the change of variables $A=pX+qY$ and $B=rX+sY$ (where $(p,q)$ and $(r,s)$ are linearly independent) to obtain an equality of the form $XY=kYX$ for some appropriate $k\ne1$. This trick had been used several times before. See, for instances, the answers to the following questions:
