Why are lines in $\mathbb{R}^3$ all congruent to one another, but circles in $\mathbb{R}^3$ are not?

Question

Lines in $\mathbb{R}^3$ are all congruent to one another, but circles in $\mathbb{R}^3$ are not all congruent to one another (because two different circles may have different radii). Visually, this is completely obvious. However, I would like a group-theoretic explanation for this.

I am thinking of $\mathbb{R}^3$ as the homogeneous space $\mathbb{R}^3 = \frac{G}{G_0} = \frac{\text{SE}(3)}{\text{SO}(3)}$, where $G = \text{SE}(3)$ is the group of (orientation-preserving) rigid motions and $G_0 = \text{SO}(3)$ is the stabilizer of the origin.

A line in $\mathbb{R}^3$ is an orbit of a point in $\mathbb{R}^3$ by a subgroup $H \leq G$ that is conjugate to the subgroup $\{ (x_1, x_2, x_3) \mapsto (x_1 + t, x_2, x_3) \colon t \in \mathbb{R}\}$ of translations by the vector $(1,0,0)$.

A circle in $\mathbb{R}^3$ is an orbit of a point in $\mathbb{R}^3$ by a subgroup $K \leq G$ that is conjugate to the subgroup $\{ (x_1 + ix_2, x_3) \mapsto (e^{i\theta}(x_1 + ix_2), x_3) \colon e^{i\theta} \in \mathbb{S}^1\}$ of rotations around the $x_3$-axis.

Two subsets $S_1, S_2$ of $\mathbb{R}^3$ are congruent if there exists $g \in \text{SE}(3)$ such that $S_2 = g \cdot S_1$.

Given these definitions of "line" and "circle" --- as orbits of subgroups --- how could we have known that all lines in $\text{SE}(3)/\text{SO}(3)$ are congruent, but not all circles in $\text{SE}(3)/\text{SO}(3)$ have this property?

In other words: What are the relevant aspects of the subgroups $H$, $K$, and $G_0$ that explain the $G$-equivalence of $H$-orbits in $G/G_0$, as opposed to the non-$G$-equivalence of all $K$-orbits in $G/G_0$?

Answer 1

Here's the general group-theoretic setup. Let $G$ be a group and $G_0,H\subset G$ be subgroups. An orbit of $H$ in $G/G_0$ can be considered as a double coset $HxG_0\subseteq G$. Let $S$ be the set of all orbits of conjugates of $H$ in $G/G_0$. Then $G$ acts on $S$ by left translation, since $g\cdot HxG_0=(gHg^{-1})gxG_0$ is a double coset for the conjugate $gHg^{-1}$.

I don't think there's any nice necessary and sufficient characterization of when $G$ acts transitively on $S$, but there are a couple simple special cases that are enough to answer your question about lines and circles.

First, suppose $H\subseteq G_0$ but some conjugate $x^{-1}Hx$ of $H$ is not contained in $G_0$. (This is true when $H$ is your $K$.) Then one element of $S$ is $HG_0=G_0$ and another is $HxG_0$. If $G$ acted transitively on $S$ there would be some $g\in G$ such that $gG_0=HxG_0$; that is, $HxG_0$ would be a left coset of $G_0$. Since $x\in HxG_0$, it would be the left coset of $x$ so $xG_0=HxG_0$. This implies $G_0=x^{-1}HxG_0$, but that is not true by assumption since $x^{-1}Hx\not\subseteq G_0$. Thus $G$ cannot act transitively on $S$.

(Interestingly, in the context of circles, this argument makes crucial use of a degenerate circle of radius $0$, which is what the double coset $HG_0=G_0$ represents. In geometric terms, it is saying that since your group $K$ fixes one point but does not fix all points, there is a circle with just one point and a circle with more than one point, and they cannot be congruent.)

Now suppose that $N(H)G_0=G$. (This is true for your line group $H$, since every translation normalizes $H$ and every rigid transformation is a composition of a rotation around the origin and a translation.) Consider any double coset $H'xG_0\in S$ for some $H'$ conjugate to $H$; we wish to show $H'xG_0$ is in the orbit of $HG_0$, so $G$ acts transitively on $S$. If $H'=gHg^{-1}$ we can first multiply $H'xG_0$ by $g^{-1}$ to assume that $H'=H$. Now by hypothesis, we can write $x=ng$ for some $n\in N(H)$ and $g\in G_0$. We then have $$HxG_0=HngG_0=HnG_0=nHG_0$$ so $HxG_0$ is indeed in the orbit of $HG_0$.

(Note that you might try and reverse this argument to prove that $N(H)G_0=G$ is actually necessary and sufficient for $G$ to act transitively on $S$. Indeed, there exists $n\in N(H)$ such that $HxG_0=nHG_0$ iff $x\in N(H)G_0$. However, this isn't quite enough to prove necessity, since you could have $HxG_0=yHG_0$ for some $y\in G$ that is not in $N(H)$, and I don't know any particularly nice way of describing when that happens. Note here that given an element of $S$, the conjugate of $H$ for which it is a double coset is not necessarily unique. See Normalizer of group action for some related discussion, and in particular the example at the end of Morgan Rodgers's answer which is one where $G$ acts transitively on $S$ but $N(H)G_0\neq G$.)

Answer 2

This is not a direct answer to your question, but an enlargment of its scope.

There exists a group which is transitive on the union of lines and circles : it is the anallagmatic group, existing in any dimension. I mention it in the third paragraph of this question and in particular i give a linear representation through generators that can be helpful for obtaining for example, given two 3D circles, the operation that will map the one onto the other.