Compare orientation between two ordered triples in $\mathbb{R}^n$

Question

Given are 2 ordered and noncollinear triples $(x_1,x_2,x_3)$ and $(y_1,y_2,y_3)$ where each $x_i$ and $y_i$ is a vector in $\mathbb{R}^n$. How can be determined whether the orientations of the triples are the same?

It is clear that the orientation depends on a reference system. Do avoid the definition of a reference system only the relative orientation between the triples shall be compared. For points in $\mathbb{R}^2$ and $\mathbb{R}^3$ we can calculate for the two sets their cross products $c_x,c_y$ and check by the dot product if $c_x,c_y$ show in the same direction.

In $\mathbb{R}^2$:

$$c_x=\begin{bmatrix} x_2^1-x_1^1\\ x_2^2-x_1^2\\ 0 \end{bmatrix} \times \begin{bmatrix} x_3^1-x_1^1\\ x_3^2-x_1^2\\ 0 \end{bmatrix} , c_y=\begin{bmatrix} y_2^1-y_1^1\\ y_2^2-y_1^2\\ 0 \end{bmatrix} \times \begin{bmatrix} y_3^1-y_1^1\\ y_3^2-y_1^2\\ 0 \end{bmatrix} $$

and in $\mathbb{R}^3$: $$c_x=\begin{bmatrix} x_2^1-x_1^1\\ x_2^2-x_1^2\\ x_2^3-x_1^3 \end{bmatrix} \times \begin{bmatrix} x_3^1-x_1^1\\ x_3^2-x_1^2\\ x_3^3-x_1^3 \end{bmatrix} , c_y=\begin{bmatrix} y_2^1-y_1^1\\ y_2^2-y_1^2\\ y_2^3-y_1^3 \end{bmatrix} \times \begin{bmatrix} y_3^1-y_1^1\\ y_3^2-y_1^2\\ y_3^3-y_1^3 \end{bmatrix} $$

\begin{align} \text{sign}\left(\langle c_x,c_y\rangle\right) \begin{cases}>0 \quad \text{same orientation}\\ <0 \quad \text{different orientation,} \end{cases} \end{align}

where $\langle c_x,c_y\rangle$ ist the dot product.

Answer 1

A $k$-frame in $\mathbb{R}^d$ is a $k$-tuple $(v_1,\cdots,v_k)$ of vectors $v_1,\cdots,v_k\in\mathbb{R}^d$ whose span has dimension $k$. If $k$ is unspecified, a "frame" in $\mathbb{R}^d$ is a $d$-frame.

The space of all $k$-frames in $\mathbb{R}^d$ is empty if $k>d$, has two components if $k=d$, and has one component of $k<d$. In other words, if $k=d$ every pair of frames is related by either an orientation-preserving or orientation-reversing transformation, whereas if $k<d$ every pair of $k$-frames is related by a rotation (an orientation-preserving transformation).

This is because we can rotate things "around" the $k$-dimensional subspace using the ambient extra $(n-k)$-dimensions. For example, in 2D, $(e_1,e_2)$ and $(e_2,e_1)$ have opposite orientations, but are related by a $180^\circ$ rotation around the axis between them in 3D, so in 3D they don't really have coordinate-independent "orientations" per se.

---

mr_e_man points out we can define the relative orientation for "most" pairs of same-dimension subspaces by orthogonally projecting from one to the other. This projection, restricted to the subspace, either preserves or reverses orientation. Indeed, the precise factor by which it distorts signed volumes (including potential negation) is calculated by the Grammian determinant, which gives the inner product of wedges. mr_e_man talks about this in his answer, and I generalize it in a previous answer here.

Answer 2

Use the wedge product instead of the cross product. https://www.av8n.com/physics/clifford-intro.htm

First let $u_1=x_1-x_3$, and $u_2=x_2-x_3$, and $v_1=y_1-y_3$, and $v_2=y_2-y_3$, to simplify the notation.

$$X=u_1\wedge u_2$$ $$Y=v_1\wedge v_2$$

The bivector $X$ represents the 2D plane containing the three points $x_1,x_2,x_3$. Similarly for $Y$.

Take the inner product of the two bivectors:

$$X^\sim=u_2\wedge u_1$$ $$X^\sim\bullet Y=(u_2\wedge u_1)\bullet(v_1\wedge v_2)$$ $$=(u_1\cdot v_1)(u_2\cdot v_2)-(u_1\cdot v_2)(u_2\cdot v_1)$$

This is positive if they have the same orientation, and negative if they have opposite orientations.