When does $P\text{SO}P^{-1} \subseteq \text{SO}$?

Question

Let $P \in \text{GL}_n^{+}(\mathbb {R})$. Suppose that $P\cdot \text{SO}(n)\cdot P^{-1} \subseteq \text{SO}(n)$.

Is it true that $P \in \lambda \text{SO}(n)$ for some $\lambda \in \mathbb{R}$?

I know that every matrix that commutes with $\text{SO}(n)$ must be in $\lambda \text{SO}(n)$ (and for $n >2$ it must be a multiple of the identity), but this is not the same question.

Edit:

In a previous version, I only required $P \in \text{GL}_n(\mathbb {R})$ instead of $\text{GL}_n^{+}(\mathbb {R})$. In that case any matrix in $\text{O}(n)$ would satisfy the requirements, so $P$ is not necessarily a multiple of special orthogonal matrix (in even dimensions).

I guess that a morally equivalent question would be to assume only $P \in \text{GL}_n(\mathbb {R})$, but to require $P\cdot \text{O}(n)\cdot P^{-1} \subseteq \text{O}(n)$. Then is it true that $P \in \lambda \text{O}(n)$ for some $\lambda \in \mathbb{R}$?

Answer 1

Changing your notation slightly, let's assume $A \in \operatorname{GL}_n^{+}(\mathbb{R})$ such that $A \cdot \operatorname{SO}_n(\mathbb{R}) \cdot A^{-1} \subseteq \operatorname{SO}_n(\mathbb{R})$. We will prove that indeed we must have $A = \lambda U$ for $\lambda > 0$ and $U \in \operatorname{SO}_n(\mathbb{R})$. First, using polar decomposition we can write $A = PU$ where $U \in \operatorname{SO}_n(\mathbb{R})$ and $P$ is positive definite. Then $$ A \cdot \operatorname{SO}_n(\mathbb{R}) \cdot A^{-1} = P \cdot U \cdot \operatorname{SO}_n(\mathbb{R}) \cdot U^{-1} \cdot P^{-1} \subseteq \operatorname{SO}_n(\mathbb{R}). $$

Since $Q \mapsto U \cdot Q \cdot U^{-1}$ is an automorphism of $\operatorname{SO}_n(\mathbb{R})$, we get $$ P \cdot \operatorname{SO}_n(\mathbb{R}) \cdot P^{-1} \subseteq \operatorname{SO}_n(\mathbb{R}). $$

Now, it is enough to show that $P = \lambda I$. Since $P$ is symmetric and positive, we can write $P = Q^{-1} \Sigma Q$ with $Q \in \operatorname{SO}_n(\mathbb{R})$ and $\Sigma$ diagonal with positive entries. Then

$$ P \cdot \operatorname{SO}_n(\mathbb{R}) \cdot P^{-1} = Q^{-1} \cdot \Sigma \cdot Q \cdot \operatorname{SO}_n(\mathbb{R}) \cdot Q^{-1} \cdot \Sigma^{-1} \cdot Q \subseteq \operatorname{SO}_n(\mathbb{R}). $$

Again, using the fact that conjugation is an automorphism, we get

$$ \Sigma \cdot \operatorname{SO}_n(\mathbb{R}) \cdot \Sigma^{-1} \subseteq \operatorname{SO}_n(\mathbb{R}). $$

Write $\Sigma = \operatorname{diag}(\lambda_1, \dots, \lambda_n)$. Assuming $n \geq 3$, the group $A_n$ of even permutations acts transitively on $\{ 1, \dots, n \}$ so for any $1 \leq i < j \leq n$ you have a special orthogonal permutation matrix $Q = Q_{i,j} \in \operatorname{SO}_n(\mathbb{R})$ which satisfies $Qe_i = e_j$. Then $(\Sigma Q \Sigma^{-1})(e_i) = \frac{\lambda_i}{\lambda_i} e_j$ has norm one iff $\lambda_i = \lambda_j$ which shows that all the diagonal entries $\lambda_i$ must be the same so $\Sigma = \lambda I$ for $\lambda > 0$. When $n = 2$, you cannot use a permutation matrix but you can choose instead a non-diagonal rotation matrix

$$ Q = \begin{pmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{pmatrix} \in \operatorname{SO}_2(\mathbb{R})$$

with $\sin \theta \neq 0$ and then $(\Sigma^{-1} Q \Sigma)(e_1) = \cos^2 \theta \cdot e_1 + \frac{\lambda_2}{\lambda_1} \sin^2 \theta \cdot e_2$ has norm one iff $\lambda_1 = \lambda_2$ which leads to the same conclusion. The remaining case $n = 1$ can be trivially verified.

The same argument shows that if $A \in \operatorname{GL}_n(\mathbb{R})$ satisfies $A \cdot \operatorname{O}_n(\mathbb{R}) \cdot A^{-1} \subseteq \operatorname{O}_n(\mathbb{R})$ then $A = \lambda U$ for $U \in \operatorname{O}_n(\mathbb{R})$ and $\lambda \neq 0$. In this case, you don't even need to handle the case $n = 2$ separately.

Answer 2

You are asking about what is the normalizer of $SO(n)$ in the general linear group.

According to groupprops, the normalizer of $O(n)$ in $GL_n(\mathbb R)$ is the orthogonal similitude group of $AA^t = \lambda I$ for all $\lambda$ nonzero. Such matrices can be rescaled to be in $O(n)$ but not necessarily $SO(n)$ if the determinant is negative.

---

ADDED:

There are two proofs at Normalizers of maximal compact groups, on mathoverflow.

I tried to fill in a few details to the first proof in the case you are interested in:

• Let $GL_n^+$ act on the set of inner products on $\mathbb R^n$. $SO(n)$ fixes precisely the scalar multiples the standard form, this set denoted by $S \cong \mathbb R_+^\times$. (This is equivalent to showing the centralizer of $SO(n)$ is scalar matrices, which I see you already did.)

• The stabilizer of $S$ is the normalizer $N$ of $SO$ in $GL^+$, and acts on $S$. This is because if $x \in SO_n$ and $g \in N$ and we consider the form $(g\_,g\_)$ then acting on this by $x$ gives $$x(g\_,g\_) = (gx\_,gx\_) = (x'g\_,x'g\_) = g.(x'\_,x'\_) = g.(\_,\_) = (g\_,g\_)$$ So $g.(\_,\_) \in S$ if $(\_,\_) \in S$. We don't need the other containment $stab(S) \subset N$.

• Finally if $A \in GL_n$ stabilizes $S$ then $A(\_,\_) = c(\_,\_)$ for some $c > 0$, and we can rescale to $A' \in S$, so $$\mathbb R_+^\times SO(n) = stab(S) \supset N$$

The second proof at that answer reduces from showing the normalizer of $SO(n)$ in $GL^+(n)$ is $\mathbb R_+^\times SO(n)$ to showing $SO(n)$ is self-normalizing in $SL(n)$, which follows trivially from being a maximal subgroup in $SL(n)$ (which is proved in a linked paper).

Answer 3

As you phrased your question, the answer is no. For $P \in O(n)$ we also have $P\cdot A \cdot P^{-1} \in SO(n)$ for all $A \in SO(n)$.