If we intuit prime (ideals) as "having nontrivial content", then units (like $\pm1$) "have no content", so shouldn't be considered prime.
Alternatively, note that when discussing prime numbers $p$, even more fundamental than FTA (unique factorization of $\mathbb{Z}$, or equivalently that the irreducible elements in $\mathbb{Z}$ coincide with the prime elements) is Euclid's lemma, the fact that any of the following equivalent conditions holds:
- $p\mid xy$ if and only if $p\mid x$ or $p\mid y$;
- If $x,y\in \mathbb{Z} \setminus p\mathbb{Z}$, then $xy\in \mathbb{Z} \setminus p\mathbb{Z}$;
- $\mathbb{Z}\setminus p\mathbb{Z}$ is a multiplicative set (which in particular requires $\mathbb{Z}/\setminus p\mathbb{Z}$ to contain the "empty product $1$", thus excluding $p=1$);
- $\mathbb{Z}/p\mathbb{Z}$ (ring of integers modulo $p$) is an integral domain (which are defined to be nonzero, thus excluding $p=1$).
The (equivalent) third and fourth conditions more generally define prime ideals in arbitrary (commutative) rings. See also Bjorn Poonen's somewhat related article on why rings should (be defined to) have identity elements.