Is this set, defined in terms of an irreducible representation, closed under inverses?

Question

Let $ \pi: H \to GL(V) $ be an irreducible representation. Define $ N_\pi^r(H) $ inductively by $$ N_\pi^{r+1}(H)=\{ g \in GL(V): g \pi(H) g^{-1} \subset N_\pi^r(H) \} $$ starting from $ N_\pi^1(H):= \pi(H) $.

Note that the set $ N_\pi^r(H) $ is generally not a group.

My question: Is $ N_\pi^r(H) $ closed under inverses?

Some thoughts: If $ \pi(H) $ is normal in $ GL(V) $ then $ N_\pi^r(H)=GL(V) $ for all $ r \geq 2 $ and the whole construction is trivial. So the question is only interesting if $ H $ is not a normal subgroup. It is certainly true for $ r=1 $ since groups are always closed under inverses.

Context:

This construction is used for a naturally occurring object called the Clifford hierarchy which appears in quantum computing. For example, the (single qubit) Clifford hierarchy is the case where $ V=\mathbb{C}^2 $ and $ H=\pi(H)= \left\langle \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}, \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix} \right\rangle $ is the tautological representation of the single qubit Pauli group.

This is a follow-up question to Is this interesting set, defined in terms of conjugation, closed under inverses? which deals with the case of general groups, not representations.

Answer 1

Maybe it is true under some restrictions on $\pi(H)$ but not in general. Here is an example. Consider $V={\mathbb R}^2$, $\pi(H)$ generated by matrices $$ \gamma_n=\left[\begin{array}{cc} 1+4n& -16 n^2\\ 1& -4n+1\end{array} \right] $$ where $n=0, 1, 2, 3,...$. The subgroup generated by these matrices is discrete, free of infinite rank, defines an irreducible faithful representation of $H=F_\infty$ to $GL(2, {\mathbb R})$. The matrix $$ \alpha= \left[\begin{array}{cc} 1& 4\\ 0& 1\end{array} \right] $$ belongs to $N^2_\pi(H)$, but $\alpha^{-1}\notin N^2_\pi(H)$. (One needs some hyperbolic geometry to prove this.)