If $G/H$ is a group under set multiplication, must $H \lhd G$?

Question

Let $G$ be a group.

Given $A,B \subseteq G$, we can define the set product $A*B= \{ab:a\in A \wedge b\in B\}. $ It turns out the the set product is associative, so that if $A,B,C \subseteq G$, we have $(A*B)*C=A*(B*C)$.

Let $H <G$. (Notation $<$ means subgroup, even possibly even equal to $G$)

Let $G/H:= \{ \{g\}*H:g\in G\}=\{ gH:g\in G\}$.

Now, I have proven that, if $H \lhd G$, then $(G/H,*)$ is a group.

I want to know if it is possible to prove the converse. Namely that, if $H<G$ and $(G/H,*)$ is a group, then $H \lhd G$.

Note that my group $(G/H,*) $ is defined slightly differently than the quotient group from most abstract algebra books, for I am not defining the product of $aH$ and $bH$ to be $abH$. (Even though, if $H \lhd G$, one can prove $abH=aH*bH).$

Answer 1

The product is actually an operation (that is, a function from $(G/H)\times (G/H)$ to $G/H$) if and only if $H$ is normal.

This is closely related to an old answer of mine. That one shows, among other things, that if you want to define the multiplication of cosets as $(aH)(bH) = (ab)H$, then this is well-defined if and only if $H$ is normal.

Basically, if $H$ is not normal, then not every set-theoretic product of left cosets is a left coset. So $*$ is not actually an operation on the set $G/H$ of left cosets. A similar argument holds for the set of right cosets.

Indeed, suppose $H$ is not normal. Then there exists $a\in G$ such that $aHa^{-1}\neq H$. In particular, there exists $h\in H$ such that $aha^{-1}\notin H$.

Now consider the product of $aH$ and $a^{-1}H$. The element $aha^{-1}e\in (aH)(a^{-1}H)$. As it does not lie in $H$, that means that if this product is to be a left coset of $H$, then it must be the left coset $aha^{-1}H$. That is, we must have $(aH)(a^{-1}H) = aha^{-1}H$.

On the other hand, the element $aea^{-1}e = e$ lies in $(aH)(a^{-1}H)$. Thus, $e$ must lie in $(aH)(a^{-1}H)$. However, the only left coset of $H$ that includes $e$ is $H$ itself. Thus, we must have $(aH)(a^{-1}H) = eH$. But this requires $aha^{-1}H = eH$, which in turn requires $aha^{-1}\in H$ . . . which is not true by assumption.

Thus, $(aH)(a^{-1}H)$ cannot be a left coset of $H$, proving that this product is not an operation on $G/H$ when $H$ is not normal.

Do take a look at that other answer for a pretty extensive discussion of this and related topics.

---

Let’s dig a bit more conceptually here about what is happening when you multiply cosets.

Lemma. Let $G$ be a group, $H$ a subgroup of $G$, and $a\in G$. If $(aH)(a^{-1}H)$ is a left coset of $H$, then it is $H$; similarly, if $(a^{-1}H)(aH)$ is a left coset of $H$, then it is $H$; and if both hold, then $a\in N_G(H)$. Conversely, if $a\in N_G(H)$, then $(aH)(a^{-1}H) =(a^{-1}H)(aH)= H$.

Proof. If $a\in N_G(H)$, then $aHa^{-1}=H$. Therefore, $(aH)(a^{-1}H) = (aHa^{-1})H = HH = H$; and since $N_G(H)$ is a subgroup, $a^{-1}\in N_G(H)$ and the same argument proves that $(a^{-1}H)(aH) = H$.

Conversely, assume that $(aH)(a^{-1}H)$ is a left coset of $H$. Since $aea^{-1}e = e\in (aH)(a^{-1}H)$, this coset must be equal to $H$. Thus, $(aH)(a^{-1}H) = H$. Therefore, for each $h\in H$ we have $aha^{-1} = aha^{-1}e \in aHa^{-1}H = H$, so $aHa^{-1}\subseteq H$. The same argument, mutatis mutandis, shows that if $(a^{-1}H)(aH)$ is a left coset, then it is $H$.

Now, if $(aH)(a^{-1}H) = H$, then for each $h\in H$ we have $aha^{-1} = aha^{-1}e \in (aH)(a^{-1}H) = H$. Thus, $aHa^{-1}\subseteq H$. Symmetrically, $(a^{-1}H)(aH) = H$ implies $a^{-1}Ha\subseteq H$, and multiplying on the left by $a$ and on the right by $a^{-1}$ we get $H\subseteq aHa^{-1}$, giving equality, so $a\in N_G(H)$. $\Box$

In particular, if $H$ is such that the product of two left cosets is always a left coset, then in particular for all $g\in G$ we have that $(gH)(g^{-1}H)$ is a left coset and $(g^{-1}H)(gH)$ is a left coset, and thus that $g\in N_G(H)$. Thus, $G\subseteq N_G(H)$, hence $H\triangleleft G$.

(Comment: I would like to weaken the hypothesis that both $(aH)(a^{-1}H)$ and $(a^{-1}H)(aH)$ are cosets to just one of them, but I’m not sure right now how to do it. The argument certainly shows that if $(aH)(a^{-1}H)$ is a coset, then it is $H$ and $aHa^{-1}\subseteq H$. Ideally of course, we should be able to say something about when the product $(aH)(bH)$ is a left coset, in terms of $a$, $b$, and the normalizer, but I haven’t fully worked that out yet . . . )

---

Note that of course one can define other operations on the set $G/H$ that have nothing to do with the multiplication of $G$ to make the set $G/H$ into a group. In greatest generality, the fact that any set can be made into a group is equivalent to the Axiom of Choice. But if you want the operation on $G/H$ to be somehow “induced” by the operation in $G$, you are pretty much forced into normal subgroups and the standard operation on the quotient.

Answer 2

Yes - if $H<G$ and $(G/H,*)$ is a group (under the definition of $*$ given), we must have $H\triangleleft G$.

We prove the contrapositive.

Suppose $H$ is a subgroup of $G$ such that $H\not\triangleleft G$, and yet $G/H$ is a group.

Given $H$'s non-normalcy, there exists an $a\in G$ such that $aHa^{-1}\not\subset H$. We'll come back to this $a$ later.

We note that the identity of $G/H$, if it exists, must be $eH=H$ itself. Suppose $G/H$ has some identity $I$.

For any $g\in G$, any element in $gH$ is of the form $gh_1$ and any element of $eH$ is some $h_2$ for $h_1,h_2\in H$, so an arbitrary element of $gH*eH$ is of the form $gh_1*h_2=g(h_1*h_2)\in gH$, i.e. $gH*eH\subset gH$.

On the other hand, $e\in eH$, so, for each $h\in H$, $gh=gh*e\in gH*eH$. Since $gh$ is an arbitrary element of $gH$, we have $gH\subset gH*eH$ which, combined with $gH*eH\subset gH$ yields $gH*eH=gH$. So $eH$ is at the very least a right identity of $G/H$.

Now, given that we assumed $G/H$ has some true identity $I$, we then get: $$I=I*eH=eH$$ confirming that $eH=H$ must be the identity of $G/H$.

Recall the fact mentioned above - that there exists an $a\in G$ and an $x\in H$ such that $aHa^{-1}\not\subset H$.

As we showed that $eH$ must be $G/H$'s identity (assuming one exists), we must have $eH*a^{-1}H=a^{-1}H$.

It is, however, immediate that $Ha^{-1}\subset eH*a^{-1}H$, as each element of $Ha^{-1}$ is of the form $ha^{-1}$ for some $h\in H$ - this element may clearly be rewritten as $(eh)*(a^{-1}e)\in eH*a^{-1}H$.

So $Ha^{-1}\subset eH*a^{-1}H=a^{-1}H$, i.e. $aHa^{-1}\subset H$, contradicting the initial assumptions.

(please comment or edit for any corrections)