$\mathcal{O}(x^n)$ stands for an “unnamed” function $f(x)$ such that there exists a positive constant $M$ so that $|f(x)|\le M|x^n|$ for $x$ sufficiently close to $0$. In particular, $f(x)$ can be a function such that $\lim_{x\to0}f(x)/x^n$ is finite.
As a consequence, $\mathcal{O}(x^n)$ is also $\mathcal{O}(x^m)$ if $m\le n$: if you can find $M$ as before, then
$$
|f(x)|\le M|x^n|=M|x^m|\,|x^{n-m}|\le M|x^m|
$$
as soon as $|x|<1$.
Also needed here is that, if $f(x)$ is $\mathcal{O}(x^n)$, then $x^kf(x)$ is $\mathcal{O}(x^{n+k})$.
By the triangle inequality, the sum of two functions that are $\mathcal{O}(x^n)$ is also a function that is $\mathcal{O}(x^n)$.
For your numerator you can write
$$
e^{x^4}=1+x^4+\mathcal{O}(x^8)
$$
and so
$$
x^2e^{x^4}=x^2+x^6+\mathcal{O}(x^{10})
$$
Therefore
$$
x^2e^{x^4}=x^2+x^6-x^2+\frac{x^6}{6}+\mathcal{O}(x^{10})=\frac{7}{6}x^6+\mathcal{O}(x^{10})
$$
The denominator is
$$
1-1+\frac{1}{2}x^6+\mathcal{O}(x^{12})=\frac{1}{2}x^6+\mathcal{O}(x^{10})
$$
Finally, you can prove that if $n>m$ and $f(x)$ is $\mathcal{O}(x^n)$, then $f(x)/x^m$ is $\mathcal{O}(x^{n-m})$.
You may want to use the “small o” notation; roughly speaking, $o(x^n)$ means you disregard terms with order larger than $x^n$. In this case
$$
x^2e^{x^4}-\sin(x^2)=
x^2(1+x^4+o(x^4))-\left(x^2-\frac{1}{6}x^6+o(x^6)\right)=\frac{7}{6}x^6+o(x^6)
$$
and
$$
1-\cos(x^3)=1-1+\frac{1}{2}x^6+o(x^6)
$$
so your limit becomes
$$
\lim_{x\to0}\frac{\frac{7}{6}x^6+o(x^6)}{\frac{1}{2}x^6+o(x^6)}
=
\lim_{x\to0}\frac{\frac{7}{6}+o(x^0)}{\frac{1}{2}+o(x^0)}
$$
and you're done because $\lim_{x\to0}o(x^0)=0$.
