I wish to find all Möbius transformations $T(z)=(az+b)/(cz+d)$ that map the circle $C=\{z\in\Bbb C:|z|=R\}$ into itself.
My attempt: Is it sufficient to find all Möbius transformations $T$ such that $|T(R)|=1$, $|T(0)|\neq 1$ and $|T(\infty)|\neq 1$ ?
I use the property that bilinear transformation maps inverse points w. r. t a circle to inverse points w. r. t. the image circle
Suppose that $w = \frac{az+b}{cz+d }$ is a transformation mapping $|z|=R$ onto $|w|=R$
Now $w=0$ , $w=\infty$ are inverse points for $|w|=R$ and they are transforms of
$z=-\frac{b}{a}, z=-\frac{d}{c}$
Respectively
$\implies -\frac{b}{a}, -\frac{d}{c}$
are inverse point for $|z|=R$
if we write $\alpha = -\frac{b}{a}$
inverse point of any point $\alpha$ W. R. T circle is $|z|=R$ is $\frac{R^2}{\bar{\alpha }}$
$ \implies \frac{-d}{c}= \frac{R^2}{\bar{\alpha }}$
So we rewrite $ w$
$w= \frac{a(z+b/a) }{c(z+d/c) }$
Using above relations
$w = \frac{a}{c} \frac{z-\alpha }{z- R^2/ \bar{\alpha }}$
$w= \frac{a\bar{\alpha }}{c}. \frac{z-\alpha }{\bar{\alpha }z-R^2}$
Let $K =\frac{a\bar{\alpha }}{c}.$
$$w = K \frac{z-\alpha }{\bar{\alpha }z-R^2}$$
Using fact $|w|=R, |z|=R$ you can easily verify $|K|=R^2$
If $T$ maps the $R$ circle into itself then $ z\mapsto {1 \over R} T(Rz)$ maps the unit circle into itself and a similar relationship holds in the opposite direction so we can assume that $R=1$.
We are looking for transformations with $ad \neq cb$ that the unit circle into itself. Note that we can assume $c=1$.
Suppose $T$ is such a transformation, then for $|z|=1$ we have $|az+b|^2=|z+d|^2$. In particular, $a \neq 0$ and $|a|^2 +|b|^2 + 2 \operatorname{re} (a\bar{b}z) = 1+|d|^2 + 2 \operatorname{re} (\bar{d}z)$ or $|a|^2 +|b|^2 + 2 \operatorname{re} ((a\bar{b}-\bar{d})z) -1 -|d|^2= 0$. Since this holds for all $|z|=1$ we must have $\bar{a}b=d$ and so $|a|^2 +|b|^2 -1 -|a|^2|b|^2 = 0$ which gives $(|a|^2-1)(|b|^2-1) = 0$.
Since $ad\neq cb$ we have $|a|^2b \neq b$ from which we get $|a|^2 \neq 1$ and so $|b| = 1$.
Hence I claim that $T$ has the form $T(z) = {az+e^{i \theta} \over z+\bar{a}e^{i \theta}}$ with $a \neq 0$ and $|a| \neq 1$.
It is straightforward to check that any such $T$ is a Möbius transformation and if $|z|=1$, then $T(z) = { 1\over e^{i \theta}z} { az + e^{i \theta} \over e^{-i\theta} + \bar{a} \bar{z}} = = { 1\over e^{i \theta}z} { az + e^{i \theta} \over \overline{ e^{i\theta} + {a} {z}} }$ and so $|T(z)| = 1$.
Hence $T$ maps the $R$ circle into itself iff $T$ has the form $T(z) = R{az+Re^{i \theta} \over z+\bar{a}Re^{i \theta}}$ with $R \neq 0$, $a \neq 0$ and $|a| \neq 1$.
