For this answer I will not rely on Cartesian coordinates, but I will not work directly in terms of latitude and longitude either. Instead, it will be convenient to represent a point with spherical coordinates $(\phi,\theta)$ as $z=e^{i\phi}\tan\dfrac\theta2$, corresponding to a stereographic projection of the sphere into the complex plane. We can now apply Mobius transformations to move the complex numbers $\alpha$, $\beta$ (corresponding to $A$, $B$) to more helpful locations.
Specifically, we can use rotations to move $\alpha$ to zero, then map $\beta$ to the real line, and finally rotate $\{\alpha,\beta\}$ to $\{r,r^{-1}\}$ on the positive real line. This amounts to rotating our coordinates so that $A$ and $B$ have zero longitude and opposite latitudes. To make this simpler, we shall assume that $\alpha$ is real without loss of generality (i.e. we take the longitude of $B$ relative to $A$).
We now state these transforms explicitly. First, note that the subgroup of the Mobius group corresponding to rotations consists of mappings of the form $z\mapsto \dfrac{u z-\overline{v}}{v z+\overline{u}}$ for complex $u,v$. A mapping of this form which takes $\alpha$ to the origin is $z\mapsto \dfrac{z-\alpha}{\alpha z+1}$; this also maps $\beta\mapsto \dfrac{\beta-\alpha}{1+\beta\alpha}= R e^{i\psi}$ for some real $R,\psi$. A rotation $z\mapsto e^{-i\phi} z$ then brings $\beta$ to $R$.
For the last transformation, consider the mapping $z\mapsto \dfrac{1+z r}{r-z}$ for real $r>0$ which takes the real line to itself. Then $\{0,R\}\mapsto \left\{\dfrac{1}{r},\dfrac{1+r R}{r-R}\right\}$; since we want the images of $0$ and $R$ to be on equal and opposite sides of the 'equator', we require their product to be unity. From this we deduce $r=R+\sqrt{1+R^2}$. Composing the three transformation gives the net rotation
$$\mathcal{T}:\;z
\mapsto \dfrac{z-\alpha}{1+\beta\alpha}
\mapsto e^{i\phi}\left(\frac{z-\alpha}{1+\beta\alpha}\right)
\mapsto \dfrac{1+e^{i\phi}\left(\frac{z-\alpha}{1+\beta\alpha}\right)r}{r-e^{i\phi}\left(\frac{z-\alpha}{1+\beta\alpha}\right)}
=\dfrac{e^{-i\phi}(1+\beta\alpha)+(z-\alpha)r}{r e^{-i\phi}(1+\beta\alpha)-(z-\alpha)}$$
We now can give a criterion for the original question: a point $z$ lies between $\alpha$ and $\beta$ if $r^{-1}\leq | \mathcal{T}(z)|r$. In principle one can do inverse transformations to map this annulus back to the original coordinates, but I won't do so explicitly. Instead I'll just summarize the result:
Let $\alpha,\beta$ be the stereographic coordinates of two points $A,B$ on the unit sphere, and define $w=R e^{i\phi}:=\dfrac{\beta-\alpha}{1+\beta\alpha}$. Then a point $C$ on the unit sphere lies between $A$ and $B$ iff its sterographic coordinate $z$ satisfies
$$\frac{1}{r}< \left|\dfrac{e^{-i\phi}(1+\beta\alpha)+(z-\alpha)r}{r e^{-i\phi}(1+\beta\alpha)-(z-\alpha)}\right|<r$$ where $r=R+\sqrt{1+R^2}$.
