Let $S = \Bbb{Z}\setminus (p)$ for a prime $p$.
Then there is a natural surjective hom $\phi :S \to (\Bbb{Z}/(p))^{\times}$ given by just taking each $x \in S$ modulo $p$.
This hom $\phi$ has a kernel which is a submonoid $M \leqslant S$. Taking cosets $sM, s \in S$ and $s M t M = stM$.
Is $S/M \approx (\Bbb{Z}/(p))^{\times}$ as groups?
For general algebraic systems, such as semigroups and monoids, the isomorphism theorems hold, but you must state them in the language of congruences, not of kernels.
In particular, even though you can define the kernel of a monoid homomorphism as the pre-image of $1$, this submonoid does not in general lead to a well-defined quotient.
For instance, take your example with $p=3$. The kernel $M$ of $\phi$ is the set of all integers that are congruent to $1$ modulo $3$, while $S$ is the set of all integers not divisible by $3$. When is $xM=yM$? Well, you need $x\in yM$, since $x1\in xM$, which means that there exists an $r\in M$ such that $yr=x$. And symmetrically, $y\in xM$ so there exists $s$ such that $y=xs$. Thus, $x=yr=xrs$ must hold in the integers. As $x\neq 0$, this means $rs=1$, so either $r=s=1$, or $r=s=-1$. Thus, $1M=-1M$ and that's the only coset they are equal to; and $2M=-2M$, and that's the only coset they are equal to. Worse, cosets may be distinct but not disjoint, as for example $2M$ contains $8$, as does $8M$, but $8M\neq 2M$ because $2\notin 8M$. So you can't really define an operation on the set of cosets the way you do with groups.
The correct notion to look at is that of "congruence".
Definition. Let $A$ be an algebra (in the sense of universal algebra: a set together with a collection of operations on $A$). A congruence on $A$ is an equivalence relation $\Phi$ on $A$ such that $\Phi$ is a subalgebra of $A\times A$, the latter endowed with coordinate-wise operations.
If you are not familiar with universal algebra, think "semigroup", "monoid", "group", "ring", "module" for algebra.
Particularizing with monoids, we have:
Theorem. Let $M$ be a monoid and let $R$ be an equivalence relation on $M$. The binary operation on equivalence classes given by $$[a]*[b]=[ab]$$ is well defined if and only if $R$ is a congruence, and in that case, $(M/R,*)$ is a monoid and $f\colon M\to M/R$ given by $f(a)=[a]$ is a monoid homomorphism.
Proof. Assume first that $R$ is congruence. Suppose $aRx$ and $bRy$; we want to show that $axRby$. We know that $(a,x)$ and $(b,y)$ are in $R$, and hence, since $R$ is a submonoid of $M\times M$, it follows that $(a,x)(b,y) = (ab,xy)\in R$. Therefore, $abRxy$, as desired. Thus $*$ is well defined.
Conversely, assume that $*$ is well-defined on equivalence classes. To show that $R$ is a submonoid of $M\times M$, note that being an equivalence relation means that $(1,1)\in R$, so the identity of $M\times M$ is in $R$; and if $(a,b)$ and $(x,y)$ are in $R$, then $[a]=[b]$ and $[x]=[y]$, hence since $*$ is well-defined we have $$[ax] = [a]*[x]=[b]*[y]=[by],$$ so $axRby$, hence $(ax,by)=(a,b)(x,y)\in R$. This shows $R$ is a submonoid of $M\times M$.
To show $M/R$ is a monoid, note that $[1]*[a]=[1a]=[a]=[a1]=[a]*[1]$, and $([x]*[y])*[z] = [xy]*[z] = [(xy)z] = [x(yz)] = [x]*[yz] = [x]*([y]*[z])$. Finally, $f(1_M) = [1_M]=[1]_{M/R}$, and $f(ab) = [ab]= [a]*[b]=f(a)f(b)$. $\Box$
Congruences of monoids are connected to morphisms:
Theorem. Let $M$ be a monoid, and let $\Phi$ be an equivalence relation on $M$. Then $\Phi$ is a congruence on $M$ if and only if there exists a monoid $N$ and a monoid homomorphism $\varphi\colon M\to N$ such that ($x\Phi y$ if and only if $\varphi(x)=\varphi(y)$).
The isomorphism theorems hold for monoids, but only when stated in terms of congruences:
First Isomorphism Theorem. Let $M$ and $N$ be monoids, and let $\varphi\colon M\to N$ be a monoid homomorphism. Then $\varphi(M)\cong M/\Phi$, where $\Phi$ is the congruence on $M$ induced by $\varphi$.
The other isomorphism theorems have congruence-variants, which I won't explain here; you can find them in any good book on Universal (or General) Algebra.
So what you want in your situation is not the kernel of $\phi$, but instead the equivalence relation on $S$ given by $$a\Phi b\iff \phi(a)=\phi(b).$$ Then $S/\Phi$ is isomorphic to $(\mathbb{Z}/(p))^{*}$ via $\overline{\phi}([a])=\phi(a)$.
The reason that we can "get away" with kernels for both groups and rings is the same reason we can "get away" with kernels to check injectivity: because if you are in a $G$, and you have a morphism $f$ defined on $G$, then $f(x)=f(y)$ if and only if $f(xy^{-1})=e$, if and only if $xy^{-1}\in\mathrm{ker}(f)$. That is, the kernel "codes" the entire congruence induced by $f$. You can find a lot more along these lines in this previous answer asking why we only use normal subgroups for quotients.
