$$\sin(u)=u-\frac{u^3}{6}+\frac{u^5}{120}+o(u^6)$$
Then, $$\sin^2(x)=(x-\frac{x^3}{6}+\frac{x^5}{120}+o(x^6))^2 =x^2-\frac{x^4}{3} +o(x^6)$$
and
$$\sin(\sin(x))=\sin\left(x-\frac{x^3}{6}+\frac{x^5}{125}+o(x^6)\right) \\=x-\frac{x^3}{6}+\frac{x^5}{120}+o(x^6) -\frac{1}{6}\left(x-\frac{x^3}{6}+\frac{x^5}{120}+o(x^6)\right)^3 + \frac{1}{120} \left(x + o(x^3)\right)^5 \\=x-\frac{x^3}{3}+\frac{x^5}{10}+o(x^6) $$
Hence
$$\frac{x\sin{(\sin{x})}-\sin^2{x}}{x^6}\\=\frac{x^2-\frac13x^4+\frac1{10}x^6-x^2+\frac13x^4-\frac2{45}x^6+o(x^6)}{x^6}=\frac1{18}+o(1)$$
Then the limit is $\frac1{18}$