To decrypt the message $\,m^e\,$ we need to take its $\,e$'th root $\bmod N.\,$ As explained here this is easy if we know a "universal period" $\,k\,$ such that $\,a^k\equiv 1,\,$ for all $\,a\,$ coprime to $\,N.\,$ Then to compute the message $\,e\,$ we can simply raise $\,m^e$ to power $\,\frac{1}{\large e} \bmod k,\,$ i.e. informally, we can take an $e$'th root by raising to power $\,\frac{1}{\large e}\,$ as usual (follow the link for a rigorous statement).
RSA uses as universal period $\,k = \phi(N)=\,$ Euler totient (for $N$ semiprime) but we could also use any multiple of $\lambda(N)=$ Carmichael lambda function = (universal) group exponent of $\,\Bbb Z_N^{*}$.
Currently the the fastest way known to compute such a universal period $\,k\,$ requires completely prime factoring $N$ then using this to compute $\phi(N)$ or $\lambda (N)$. Since this is not practical for a large modulus $\,N,\,$ RSA powering encryption is a good one-way (trapdoor) function for cryptography.
So, to answer your question, given what is currently known, generally there is no fast way to decrypt (RSA) powered messages without a complete factorization of $N$.