In number theory, "$\pmod n$" usually isn't an operation. It's most often part of $\equiv$. So, for instance, $5\equiv 2\pmod 3$ doesn't mean that if you apply a "modulus 3 operation" to $5$, you get $2$. It means that $5$ and $2$ are congruent modulo $3$. In other words, that they are some multiple of $3$ apart.
So there is no problem at all with saying $5\equiv 8\pmod 3$, or $5\equiv-4\pmod3$.
We also get that $\pmod n$ and $\pmod{{-}n}$ are equivalent, so we usually don't use negative modulus, because it doesn't give us anything at all.
As for the %
(modulo) operator in programming languages, and on calculators, how they behave with negative numbers is partially up to whoever designed it. There are several equally correct ways to do it. Some may have opinions on which way is the "best", I personally haven't used them enough to make up my mind.