The limit to be found is
$$ \lim_{x\to 0}\frac{\sin(x)\arcsin(x)-x^2}{x^6}$$
I've tried l'hopital rule but it gets really messy. I've also tried splitting it into 2 limits but that doesn't work. I can't think of any meaningful substitution either.
PS: I know the answer $1/18$ but I'm interested in the method. Thank you.
If you know that the answer is $\frac 1 {18}$, there is probably a typo in the expression.
Any way consider $$\dfrac{\sin (x)\sin^{-1} (x^{2})}{x^{6}}=\frac 1 {x^3} \times \frac {\sin(x)} {x}\times \frac {\sin^{-1}(x^2)} {x^2}$$ and remember how behave the second and third term.
Edit
Now, the problem is $$\frac{\sin(x)\arcsin(x)-x^2}{x^6}$$ So, just as Workaholic commented, use the classical Taylor series $$\sin(x)=x-\frac{x^3}{6}+\frac{x^5}{120}+O\left(x^6\right)$$ $$\sin^{-1}(x)=x+\frac{x^3}{6}+\frac{3 x^5}{40}+O\left(x^6\right)$$ Making the product $$\sin(x)\sin^{-1}(x)=x^2+\frac{x^6}{18}+O\left(x^7\right)$$
Use standard equivalents: $\sin x\sim_0 x,\quad\arcsin u\sim_0 u$, hence $$\frac{\sin x\arcsin x^2}{x^6}\sim_0\frac{x\cdot x^2}{x^6}=\frac1{x^3}$$ Thus $\;\displaystyle\lim_{x\to 0^+}\frac{\sin x\arcsin x^2}{x^6}=+\infty$, $\quad\displaystyle\lim_{x\to 0^-}\frac{\sin x\arcsin x^2}{x^6}=-\infty$.
NOTE.-This answer was for the question before which is distinct of the given now
It is well known that in a circle of radius $r$ the arc $\widehat{AB}$ subtended by an angle $x$ is given by $\widehat {AB}=rx$. In the unitary circle it is also known that $\sin x\approx x$ for small values so in the figure $\overline{BB'}\approx \widehat{AB}$ for small values. From this it is deduced that, for all natural n (in particular for $n=2$) $$\lim\limits_{x\to 0} \frac{\arcsin x^n}{x^n}=1 $$
Writing your expression as the product $$\frac{\sin x}{x}\cdot \frac{\arcsin x^2}{x^2}\cdot \frac{1}{x^3} $$ you can see it is not possible your limit be $\frac{1}{18}$ (It does not exist because "tends to" $\pm\infty$).I conclude that your $\frac {1}{18}$ as answer is a typo.
Some basic limits can be obtained by repeated use of l'Hospital's Rule easily, \begin{eqnarray*} \lim_{x\rightarrow 0}\frac{\sin x-x}{x^{3}} &=&-\frac{1}{6},\ \ \ \ \ \ \ \ and\ \ \ \ \ \ \ \ \ \lim_{x\rightarrow 0}\frac{\arcsin x-x}{x^{3}}=\frac{1}{% 6} \\ \lim_{x\rightarrow 0}\frac{\sin x-x+\frac{1}{6}x^{3}}{x^{5}} &=&\frac{1}{120}% ,\ \ \ \ \ \ \ \ and\ \ \ \ \ \ \ \ \ \lim_{x\rightarrow 0}\frac{\arcsin x-x-% \frac{1}{6}x^{3}}{x^{5}}=\frac{3}{40} \end{eqnarray*} Now, re-write the original expression as follows:
\begin{eqnarray*} \frac{\sin x\arcsin x-x^{2}}{x^{6}} &=&\frac{(\left[ \sin x-x\right] +x)([\arcsin x-x]+x)-x^{2}}{x^{6}} \\ &=&\frac{\left[ \sin x-x\right] [\arcsin x-x]+x\left[ \sin x-x\right] +x[\arcsin x-x]+x^{2}-x^{2}}{x^{6}} \\ &=&\frac{\left[ \sin x-x\right] [\arcsin x-x]}{x^{6}}+\frac{x\left[ \sin x-x+% \frac{1}{6}x^{3}\right] +x[\arcsin x-x-\frac{1}{6}x^{3}]}{x^{6}} \\ &=&\frac{\left[ \sin x-x\right] }{x^{3}}\frac{[\arcsin x-x]}{x^{3}}+\frac{% \left[ \sin x-x+\frac{1}{6}x^{3}\right] }{x^{5}}+\frac{[\arcsin x-x-\frac{1}{% 6}x^{3}]}{x^{5}} \end{eqnarray*} It follows that \begin{eqnarray*} \lim_{x\rightarrow 0}\frac{\sin x\arcsin x-x^{2}}{x^{6}} &=&\lim_{x% \rightarrow 0}\frac{\left[ \sin x-x\right] }{x^{3}}\lim_{x\rightarrow 0}% \frac{[\arcsin x-x]}{x^{3}}+\lim_{x\rightarrow 0}\frac{\left[ \sin x-x+\frac{% 1}{6}x^{3}\right] }{x^{5}}+\lim_{x\rightarrow 0}\frac{[\arcsin x-x-\frac{1}{6% }x^{3}]}{x^{5}} \\ &=&\left( -\frac{1}{6}\right) \left( \frac{1}{6}\right) +\left( \frac{1}{120}% \right) +\left( \frac{3}{40}\right) \\ &=&\frac{1}{18}. \end{eqnarray*}
For completeness I post my approach (mentioned in comments to the question). Using L'Hospital's Rule we can see that $$\lim_{x \to 0}\frac{\sin x - x}{x^{3}} = \lim_{x \to 0}\frac{\cos x - 1}{3x^{2}} = -\frac{1}{3}\lim_{x \to 0}\frac{1 - \cos^{2}x}{x^{2}(1 + \cos x)} = -\frac{1}{6}\lim_{x \to 0}\frac{\sin^{2}x}{x^{2}} = -\frac{1}{6}\tag{1}$$ And therefore $$\lim_{x \to 0}\frac{\sin x - x}{\sin^{3}x} = \lim_{x \to 0}\frac{\sin x - x}{x^{3}}\cdot\frac{x^{3}}{\sin^{3}x} = -\frac{1}{6}$$ Putting $\sin x = t$ so that $x = \arcsin t$ we get $$\lim_{t \to 0}\frac{t - \arcsin t}{t^{3}} = -\frac{1}{6}$$ or $$\lim_{x \to 0}\frac{\arcsin x - x}{x^{3}} = \frac{1}{6}\tag{2}$$ Multiplying equations $(1)$ and $(2)$ we get $$\lim_{x \to 0}\frac{(\sin x - x)(\arcsin x - x)}{x^{6}} = -\frac{1}{36}$$ or $$\lim_{x \to 0}\frac{\sin x\arcsin x - x(\sin x + \arcsin x) + x^{2}}{x^{6}} = -\frac{1}{36}$$ or $$\lim_{x \to 0}\frac{\sin x\arcsin x - x^{2} - x(\sin x + \arcsin x) + 2x^{2}}{x^{6}} = -\frac{1}{36}$$ or $$\lim_{x \to 0}\frac{\sin x\arcsin x - x^{2}}{x^{6}} + \lim_{x \to 0}\frac{2x - \sin x - \arcsin x}{x^{5}} = -\frac{1}{36}$$ or $$A + B = -\frac{1}{36}\tag{3}$$ where $A$ is the limit asked in question and we calculate the limit $B$ as follows: \begin{align} B &= \lim_{x \to 0}\frac{2x - \sin x - \arcsin x}{x^{5}}\notag\\ &= \lim_{x \to 0}\dfrac{2 - \cos x - \dfrac{1}{\sqrt{1 - x^{2}}}}{5x^{4}}\text{ (via L'Hospital's Rule)}\notag\\ &= \frac{1}{5}\lim_{x \to 0}\frac{(2 - \cos x)\sqrt{1 - x^{2}} - 1}{x^{4}\sqrt{1 - x^{2}}}\notag\\ &= \frac{1}{5}\lim_{x \to 0}\frac{(2 - \cos x)\sqrt{1 - x^{2}} - 1}{x^{4}}\notag\\ &= \frac{1}{5}\lim_{x \to 0}\frac{(2 - \cos x)^{2}(1 - x^{2}) - 1}{x^{4}\{(2 - \cos x)\sqrt{1 - x^{2}} + 1\}}\notag\\ &= \frac{1}{10}\lim_{x \to 0}\frac{(2 - \cos x)^{2}(1 - x^{2}) - 1}{x^{4}}\notag\\ &= \frac{1}{10}\lim_{x \to 0}\frac{(1 + t)^{2}(1 - x^{2}) - 1}{x^{4}}\text{ (puting }t = 1 - \cos x)\notag\\ &= \text{(note that }t/x^{2} \to 1/2\text{ as }x \to 0)\notag\\ &= \frac{1}{10}\lim_{x \to 0}\frac{2t + t^{2} - x^{2} - 2tx^{2} - t^{2}x^{2}}{x^{4}}\notag\\ &= \frac{1}{10}\lim_{x \to 0}\left(\frac{2t - x^{2}}{x^{4}} + \frac{t}{x^{2}}\cdot\frac{t}{x^{2}} - 2\cdot\frac{t}{x^{2}} - \frac{t}{x^{2}}\cdot\frac{t}{x^{2}}\cdot x^{2}\right)\notag\\ &= \frac{1}{10}\lim_{x \to 0}\left(\frac{2t - x^{2}}{x^{4}} + \frac{1}{2}\cdot\frac{1}{2} - 2\cdot\frac{1}{2} - \frac{1}{2}\cdot\frac{1}{2}\cdot 0\right)\notag\\ &= \frac{1}{10}\lim_{x \to 0}\frac{2t - x^{2}}{x^{4}} - \frac{3}{40}\notag\\ &= \frac{1}{10}\lim_{x \to 0}\frac{4\sin^{2}(x/2) - x^{2}}{x^{4}} - \frac{3}{40}\notag\\ &= \frac{1}{10}\lim_{u \to 0}\frac{4\sin^{2}u - 4u^{2}}{16u^{4}} - \frac{3}{40}\text{ (putting }x = 2u)\notag\\ &= \frac{1}{40}\lim_{u \to 0}\frac{\sin^{2}u - u^{2}}{u^{4}} - \frac{3}{40}\notag\\ &= \frac{1}{40}\lim_{u \to 0}\frac{\sin u - u}{u^{3}}\cdot\frac{\sin u + u}{u} - \frac{3}{40}\notag\\ &= \frac{1}{40}\lim_{u \to 0}\frac{-1}{6}\cdot\left(\frac{\sin u}{u} + 1\right) - \frac{3}{40}\text{ (from equation (1))}\notag\\ &= -\frac{1}{120} - \frac{3}{40} = -\frac{1}{12}\notag \end{align} From equation $(3)$ we get the desired limit $$A = -\frac{1}{36} - B = -\frac{1}{36} + \frac{1}{12} = \frac{1}{18}$$
