Check if subgroup is normal based on generators

Question

Let's say we have a group $G=\langle g_1,g_2,g_3\rangle$ for some generators $g_i$. Further, suppose I have some subgroup $\langle h_1,h_2\rangle =H\subset G$ for some generators $h_i$. For $H$ to be normal in $G$ we must have $gHg^{-1}=H$ for all $g\in G$. Is there an easy way to check this computationally? What I mean is, can I do something like check $g_jh_ig_j^{-1}\in H$ for each $i,j$? Or is this not sufficient? If it is sufficient, is there something even easier we can do, like check if each $g_jh_ig_j^{-1}$ is one of the generators $h_k$?

Answer 1

Theorem. Let $G$ be a group, and $N\leq G$ a subgroup. Let $X\subseteq N$ and $Y\subseteq G$ be subsets such that $N=\langle X\rangle$ and $G=\langle Y\rangle$. The following are equivalent:

1. $N\triangleleft G$.
2. For all $g\in G$, $gNg^{-1}=N$.
3. For all $g\in G$, $gNg^{-1}\subseteq N$.
4. For all $g\in G$ and $n\in N$, $gng^{-1}\in N$.
5. For all $g\in G$ and all $x\in X$, $gxg^{-1}\in N$.
6. For all $y\in Y$, $yNy^{-1}=N$.
7. For all $y\in Y$, $yNy^{-1}\subseteq N$ and $y^{-1}Ny\subseteq N$.
8. For all $y\in Y$ and $x\in X$, $yxy^{-1}\in N$ and $y^{-1}xy\in N$.

Proof. (1)$\iff$(2) is the usual definition.

Clearly (2)$\implies$(3); to get (3)$\implies$(2), note that for $g\in G$, we have $gNg^{-1}\subseteq N$ from (3), and applying $3$ to $g^{-1}$ we have $g^{-1}Ng\subseteq N$. Left multiplying by $g$ and right multiplying by $g^{-1}$ we get $N\subseteq gNg^{-1}$, giving $gNg^{-1}=N$, thus proving (2).

It is easy to see that (3)$\iff$(4).

(4)$\implies$(5) is immediate. To get (5)$\implies$(4), first note that for all $x\in X$ and $g\in G$, since $gxg^{-1}\in N$ and $N$ is a subgroup, then $gx^{-1}g^{-1}=(gxg^{-1})^{-1}\in N$. Now let $n\in N$. Since $N=\langle X\rangle$, we can write $n = x_1^{\epsilon_1}\cdots x_n^{\epsilon_n}$, where $x_i\in X$ are not necessarily distinct, and $\epsilon_i\in\{1,-1\}$ for all $i$. Then $$\begin{align*} gng^{-1} &= g(x_1^{\epsilon_1}\cdots x_n^{\epsilon_n})g^{-1}\\ &= (gx_1^{\epsilon_1}g^{-1})(gx_2^{\epsilon_2}g^{-1})\cdots(gx_n^{\epsilon_n}g^{-1}). \end{align*}$$ Each $gx_i^{\epsilon_i}g^{-1}\in N$, so this is a product of elements of $N$, and hence lies in $N$. Thus, $gng^{-1}\in N$, proving (4).

(2)$\implies$(6) is immediate. To prove that (6)$\implies$(2), note that from $yNy^{-1}=N$ we get, left multiplying by $y^{-1}$ and right multiplying by $y$, that $N=y^{-1}Ny$. Let $g\in G$; using $G=\langle Y\rangle$, we know that we can write $g=y_1^{\epsilon_1}\cdots y_n^{\epsilon_n}$, with $y_i\in Y$ and $\epsilon_i\in \{1,-1\}$ for all $i$. We also know that $y_j^{\epsilon_j}Ny_j^{-\epsilon_j}=N$. From this, we get $$\begin{align*} gNg^{-1} &= (y_1^{\epsilon_1}\cdots y_n^{\epsilon_n})N(y_1^{\epsilon_1}\cdots y_n^{\epsilon_n})^{-1}\\ &= y_1^{\epsilon_1}\cdots y_{n-1}^{\epsilon_{n-1}}(y_n^{\epsilon_n }Ny_n^{-\epsilon_1})y_{n-1}^{-\epsilon_{n-1}}\cdots y_1^{-\epsilon_1}\\ &= y_1^{\epsilon_1}\cdots y_{n-1}^{\epsilon_{n-1}}Ny_{n-1}^{-\epsilon_{n-1}}\cdots y_1^{-\epsilon_1} \end{align*}$$ and inductively we get $gNg^{-1}=N$, proving 2.

(6)$\iff$(7), since $y^{-1}Ny\subseteq N$ is equivalent, multiplying by $y$ on the left and $y^{-1}$ on the right, to $N\subseteq yNy^{-1}$.

(7)$\implies$(8) is clear. For (8)$\implies$(7), using the same argument to show that (5)$\implies$(4) we see that (8) implies that $yny^{-1}\in N$ and $y^{-1}ny\in N$ for all $n\in N$, giving (7). $\Box$

However, in (7) and (8) it is not, in general, enough to require $yNy^{-1}\subseteq N$ (respectively $yxy^{-1}\in N$). A counterexample can be found in this post.