7

Let $ G $ be a group. Let $ H $ be a finite subgroup. Define $ N_G^r(H) $ inductively by $$ N_G^{r+1}(H)=\{ g \in G: g H g^{-1} \subset N_G^r(H) \} $$ starting from $ N_G^1(H):= H $.

Note that $ N^2_G(H) = N_G(H) $ is just the normalizer in $ G $ of $ H $. Although $ N_G^1(H)= H $ and $ N_G^2(H)= N_G(H) $ are both groups, the set $ N_G^r(H) $ is generally not a group for $ r \geq 3 $.

My question: Is $ N_G^r(H) $ closed under inverses?

Some thoughts: If $ H $ is normal in $ G $ then $ N_G^r(H)=G $ for all $ r \geq 2 $ and the whole construction is trivial. So the question is only interesting if $ H $ is not a normal subgroup. It is certainly true for $ r=1,2 $ since groups are always closed under inverses.

Context:

This construction is used for a naturally occurring object called the Clifford hierarchy which appears in quantum computing. For example, the (single qubit) Clifford hierarchy is the case where $ G=\mathrm{U}(2) $ and $ H= \left\langle \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}, \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix} \right\rangle $ is the single qubit Pauli group.

Update: As pointed out by Arturo Magidin in The set of all $x$ such that $xHx^{-1}\subseteq H$ is a subgroup, when $H\leq G$ if we allow $ H $ to be infinite then even $ N^2_G(H) $ is not necessarily closed under inverses. See the example given for $ G=GL(2,\mathbb{Q}) $ and $ H = \left\{\left.\left(\begin{array}{cc} 1 & m\\ 0 & 1\end{array}\right)\in G\ \right|\ m\in\mathbb{Z}\right\} $ where $ x = \left(\begin{array}{cc} 2 & 0\\ 0 & 1\end{array}\right) $ is in $ N^2_G(H) $ but $ x^{-1} $ is not.

Ian Gershon Teixeira
  • 8,010
  • 3
    It is not even true that $N_G^2(H)$ is necessarily the normalizer, at least not for infinite groups. $gHg^{-1}\subseteq H$ does not imply $g^{-1}Hg\subseteq H$ in the infinite case. See here. – Arturo Magidin Nov 14 '23 at 21:41
  • @ArturoMagidin wow fabulous counterexample! You're right I'm used to the case that $ H $ is finite. I've updated my question to add that assumption. – Ian Gershon Teixeira Nov 14 '23 at 22:11

1 Answers1

9

I was immediately convinced that the answer to this question had to be no, because there is no conceivable reason why it should be yes. So then it just became a matter of systematically searching for a counterexample, and fortunately one can use computers to do that. I just tried all the subgroups of $S_n$ for increasing $n$, and found an example with $n=8$.

To be specific, $H$ is the Klein 4-group $$ H := \langle (1,2)(3,4)(5,6)(7,8), (1,3)(2,4)(5,7)(6,8) \rangle.$$

Then $N := N_G(H)$ (with $G=S_8$) has order $192$, and there are $5760$ elements $g \in G$ satisfying $g H g^{-1} < N$. In particular, the element $g := (1, 8, 2, 5, 7, 6, 3, 4)$ has this property, but $g^{-1}$ does not (composing permutations from left to right.)

Derek Holt
  • 90,008
  • 1
    Is it possible that the answer is different if $ H $ is an irreducible subgroup of a unitary group $ U(d) $? I'm just asking because in this example $ H $ is reducible as a subgroup of $ U(8) $ whereas the Pauli group $ P_n $, which is the motivation for this question, is irreducible as a subgroup of $ U(2^n) $. (also just fyi I tried to award the bounty but it says I have to wait another 2 hours) – Ian Gershon Teixeira Nov 20 '23 at 14:42
  • 1
    That would be a separate question, but I would still guess that the answer was no. – Derek Holt Nov 20 '23 at 20:24
  • You were right irreducibility alone isn't enough! https://math.stackexchange.com/a/4819537/758507 But I'm still holding out hope that irreducibility+finiteness will be enough: https://mathoverflow.net/questions/459505/is-this-set-defined-in-terms-of-an-irreducible-representation-closed-under-inv – Ian Gershon Teixeira Dec 03 '23 at 20:21