The first equation $a^{-1}b^2a=b^3 \Rightarrow a^{-1}b^{2k}a=b^{3k}$ for all $k \in {\mathbb Z}$, so
$$a^{-2}b^{-4}a^2 = a^{-1}b^{-6}a = b^{-9}.$$
The second equation gives $a^{-2}b^{-1}a^2=ab^{-1}$, so
$$a^{-2}b^{-4}a^2 = (ab^{-1})^4$$ and hence
$$b^{-9} = (ab^{-1})^4.$$
Since $b^9$ commutes with $b$ and $(ab^{-1})^4$ commutes with $ab^{-1}$, it follows that $b^{-9} = (ab^{-1})^4 \in Z(G)$.
So $N = \langle b^9 \rangle \le Z(G)$. In the quotient $G/N$, the image of $b^4$ is conjugate to $b^9$, which is trivial, so the image of $b$ is trivial, hence the image of $a^4$ is trivial, but $a^4$ is conjugate to $a^9$, so the image of $a$ is trivial.
Hence $G/N$ has order $1$, so $G=N \le Z(G)$ and hence $G$ is abelian and it is easily proved that $G$ is trivial.