$$\lim_{x\to 0} \frac{x^2 \frac{\sin bx}{bx} (bx)}{x(\frac{ax}{x}-\frac{\sin x}{x})}$$ $$=\lim_{x\to 0} \frac{bx^2 \frac{\sin bx}{bx}}{a-\frac{\sin x}{x}}$$ $$=0$$
Which is obviously wrong. I think I am suffering from lack of conceptual clarity in limits, so what exactly is going wrong in this question?
The limit in your second step gives zero only for $a\neq 1$, therefore we need $a=1$ as a condition.
We can proceed as follows, since we have that $b\neq 0$ and
$$\frac{x^2 \sin (bx)}{ax-\sin x}=\frac{\sin bx}{bx}\frac{bx^3}{ax-\sin x}$$
and since $\frac{\sin bx}{bx} \to 1$ we need $\frac{bx^3}{ax-\sin x} \to 1$ which implies $a=1$ and finally we obtain
$$b=\lim_{x \to 0}\frac{x-\sin x}{x^3}$$
which can be easily found by l'Hopital or by Taylor's expansion.
Hint:
$$\dfrac{x^2\sin bx}{ax-\sin x}=b\cdot\dfrac{\sin bx}{bx}\cdot\dfrac1{\dfrac{a-1}{x^2}+\dfrac{x-\sin x}{x^3}}$$
Now using Are all limits solvable without L'Hôpital Rule or Series Expansion
$a-1$ must be $=0$
$$L=\lim_{x\to 0} \frac{x^2\sin bx}{ax-\sin x}=\lim_{x\to 0} \frac{x^2(bx-b^3x^3/6+...)}{ax-x+x^3/6}=\lim_{x\to 0} \frac{bx^3-b^5 x^5/6}{ax-x+x^3/6}$$ If $L=1$, then $a=1,b=1/6.$
