While bounds on the number of isomorphism classes of groups of order $p^n$ where $p$ is prime have been known for quite a while (such as the work of Higman$^{[1]}$ and Sims$^{[2]}$) which give us the bounds for $f(n,p)$ (which returns the number of isomorphism classes of groups of order $p^n$) as follows:
$f(n,p) = p^{An^3} \operatorname{where} A(n,p) = \frac{2}{27} + O(n^{-1/3})$
However, while this tells us that for a fixed $p$ the value of $f(n,p)$ grows very rapidly with n, when I've been reading through the papers I don't think I have enough knowledge to understand why exactly they should grow this rapidly - for example, the number of abelian groups of order $p^n$ is simply the number of partitions of $n$, and while this number grows quite rapidly with n, this number is tiny in comparinson to the number of non-abelian groups of the same exponent.
Therefore, would it be possible for someone to try and explain to me some reasons for why there are so many groups of order $p^n$ for $n>2$ (as it is fairly straightforward to categorize such groups for $n=1$ and $2$), and also why there are so many such $p$-groups when compared to groups of similar sized order? (I suspect these two questions are closely linked in some respects, which is why I ask them both, although answers on one or the other are equally appreciated)
For those who would like them, the references to the papers of Higman and Sims are below:
$[1]$ - Proc. London Math. Soc. (1960) s3-10 (1): 24-30. doi: 10.1112/plms/s3-10.1.24
$[2]$ - Proc. London Math. Soc. (1965) s3-15 (1): 151-166. doi: 10.1112/plms/s3-15.1.151