$\newcommand{\Matrix}[1]{\left(\begin{matrix}#1\end{matrix}\right)}$
The perpendicular angular distance $α$ of point P from the great circle through points Q and R is
$$ \sin(α) = \frac{\hat{P} ⋅ (\hat{Q} ×
\hat{R})}{|\hat{Q} × \hat{R}|} $$
where $⋅$ denotes the inner product, $×$ the vector cross product, $\hat{v}$ indicates an arbitrary vector with length 1, $|\vec{v}|$ stands for the length of an arbitrary vector $\vec{v}$, and $\hat{P}$, $\hat{Q}$, $\hat{R}$ stand for vectors of length 1 pointing from the center of the sphere to points P, Q, and R.
The inner product of two vectors is
$$ \Matrix{x_1 \\ y_1 \\ z_1} ⋅ \Matrix{x_2 \\ y_2 \\
z_2} = x_1x_2 + y_1y_2 + z_1z_2 $$
and the vector cross product is
$$ \Matrix{x_1 \\ y_1 \\ z_1} × \Matrix{x_2 \\ y_2 \\
z_2} = \Matrix{y_1z_2 − y_2z_1 \\ z_1x_2 − z_2x_1 \\ x_1y_2 − x_2y_1} $$
The polar coordinates $θ$ (longitude) and $φ$ (latitude) are converted to cartesian coordinates $x,y,z$ through
\begin{eqnarray}
x & = & \cos(θ)\cos(φ)
\\ y & = & \sin(θ)\cos(φ)
\\ z & = & \sin(φ)
\end{eqnarray}
The derivation of the first formula is as follows:
The vector cross product $$\vec{S} = \hat{Q} × \hat{R}$$ is by definition perpendicular to both $\hat{Q}$ and $\hat{R}$, hence perpendicular to the plane of the great circle. Its length is not necessarily equal to 1, so we mark it as an ordinary vector $\vec{S}$, not a unit vector $\hat{S}$.
Let $β$ be the angle between $\hat{P}$ and $\vec{S}$. Then, using the definition of the inner product, $$\hat{P}⋅\vec{S} = |\hat{P}| |\vec{S}| \cos(β) = |\vec{S}| \cos(β)$$ so $$\cos(β) = \frac{\hat{P}⋅\vec{S}}{|\vec{S}|}$$
The angle $β$ between $\vec{S}$ and $\hat{P}$ and the angle $α$ between $\hat{P}$ and the plane of the great circle add up to 90°, which is the angle between $\vec{S}$ and the plane of the great circle, so $$\cos(β) = \sin(α)$$
Combined, this yields the first equation.
The sign of $\sin(α)$ depends on the order of $\hat{Q}$ and $\hat{R}$ in the vector cross product. If you swap $\hat{Q}$ and $\hat{R}$, then $\sin(α)$ and $\vec{S}$ get multiplied by −1.
Let $z_S$ be the z coordinate of $\vec{S}$. Then P is north of the great circle through Q and R if $z_S\sin(α) \gt 0$, and south of the great circle if $z_S\sin(α) \lt 0$
