Any automorphism of a group $G$ is a bijection that fixes the identity, so an easy upper bound for the size of $\operatorname{Aut}(G)$ for a finite group $G$ is given by
\begin{align*}\lvert\operatorname{Aut}(G)\rvert \leq (|G| - 1)! \end{align*}
This inequality is an equality for cyclic groups of orders $1$, $2$ and $3$ and also the Klein four-group $\mathbb{Z}_2 \times \mathbb{Z_2}$. I think it's reasonable to believe that they are the only groups with this property. The factorial $(|G| - 1)!$ is eventually huge. I searched through groups of order less than $100$ with GAP and found no other examples.
The problem can be reduced to the abelian case. We can check the groups of order $< 6$ by hand. Then if $|G| \geq 6$ and the equality holds, we have $\operatorname{Aut}(G) \cong S_{|G|-1}$. Now $\operatorname{Inn}(G)$ is a normal subgroup of $\operatorname{Aut(G)}$, and is thus isomorphic to $\{(1)\}$, $A_{|G|-1}$ or $S_{|G|-1}$. This is because $A_n$ is the only proper nontrivial normal subgroup of $S_n$ when $n \geq 5$. We can see that $(|G| - 1)!/2 > |G|$ and thus $\operatorname{Inn}(G) \cong G/Z(G)$ is trivial.
How to prove that there are no other groups for which the equality $\lvert\operatorname{Aut}(G)\rvert = (|G| - 1)!$ holds? Are any better upper bounds known for larger groups?