rotating linear dependent vectors in space

Question

I'm not quite sure how to write this succinctly with mathematical symbols, so I just had to write it out in english. Any edit to suggest how to write it in mathematical form would be appreciated even if it does not provide a full answer.

If you have a set of linearly dependent vectors in some space in $N$ dimensions, then that set of vectors can be thought of existing in a $N-1$ dimensional subspace of the space that the vectors exist in. (eg: In 3 dimensions a set of vectors that are linearly dependent on one another exist on a plane, a 2 dimensional subspace, in that 3 space)

If you apply a rotation matrix to the a matrix of linearly depended vectors then it makes sense to me that you could rotate this subspace so that one of the vector components is zero for all of the vectors in that matrix, producing a zero column.

This method makes sense as a zero row in a square matrix would give a determinant of zero as row reduction gives a zero row. The thing I want to know is have I miraculously come to the right conclusion through wrong methods and therefore the method will not work in some cases, or is the method sound and work for all cases described?

Answer 1

Your method is perfectly valid. You are saying a plane in 3d-space containing origin can be rotated to become one of the co-ordinate planes. Think of your plane as a door that is ajar. By shutting the door (i.e. rotating about the axis containing the keel/hinges) it becomes a plane where one co-ordinate is zero. You can apply this rotation operation (around arbitrary axis)to make any plane to coincide with other (both should contain the origin). This intuition is obtained from our daily interaction with space. What is important is when we learn mathematics in formal symbols and notation not to lose track of this grounding.

To use fancy language, the orthogonal group acts transitively on the grassmannian, that is what your statement is expressed in higher mathematics.

Answer 2

Suppose one of the columns of an $N\times N$ matrix is a linear combination of the others. Discard it, getting an $N\times(N-1)$ matrix. And keep discarding such dependent columns until non are left, so you have an $N\times K$ matrix $A$ with independent columns and $K<N$.

Now consider the matrix $P=A(A^T A)^{-1}A^T$. If $A$ were a nonsingular square matrix than $P$ would be the identity matrix, but $P$ has rank $K<N$. It behaves like the identity matrix is this respect: If $x$ is member of the column space of $A$ then $Px=x$. That's easy to show: If $x$ is in the column space of $A$ then $x=Aw$ for some $K\times 1$ vector $w$. So $Px = \Big(A(A^T A)^{-1} A^T\Big)(Aw)$. Do the routine simplifications and this comes down to $x$.

But if $x$ is orthogonal to the column space of $A$, then $Px=0$. That's even easier to show: $Px = \Big(A (A^T A)^{-1} A^T\Big) x = A (A^T A)^{-1}\Big(A^Tx\Big)$, and that last factor, $A^Tx$, is $0$.

Notice that $P$ is symmetric: $P^T=P$. The spectral theorem says a real symmetric matrix can be diagonalized by some rotation matrix $G$, i.e. $GG^T=G^TG=I$ and $\det G=1$. So you have $$ P = GDG^T = G\begin{bmatrix} d_1 \\ & d_2 \\ & & d_3 \\ & & & \ddots \end{bmatrix} G^T. $$ What are those diagonal entries? Since $P^2=P$ (check that) we must have $d_i^2=d_i$, so $d_i=\text{either 0 or 1}$. Some are $1$ (exactly $K$ of them) and the rest are $0$ (exactly $N-K$ of them).