Let $A$ be a $3\times 3$ orthogonal matrix with $\det A =1 $, whose angle of rotation is different from $0$ or $\pi$, and let $ M = A -A^t$
-Show that $M$ has rank 2, and that a nonzero vector $X$ in the nullspace of $M$ is an eigenvector of $A$ with eigenvalue 1.
-Find such an eigenvector explicitly interms of the entries of the matrix $A$.
geometrically, $Mx$ is rotating $x$ in one direction, then the opposite, and taking the difference. this difference is always zero on the axis of rotation (and only on the axis of rotation since the angle is different from $0, \pi$):
if $Mx=0$ then $A^2x=x$, i.e. rotating by twice some angle brings you back to where you started. since the angle of rotation is not $0, \pi$, the only way for that to happen is if $x$ is on the axis of rotation, i.e. $x$ is an eigenvector with eigenvalue $1$.
try finding the axis of rotation in terms of the coordinates yourself.
Since $A$ is orthogonal (real entries) and det($A$)=1, one eigenvalue of $A$ is 1 and the other two are $e^{i\theta}$ and $e^{-i\theta}$ ($\theta \neq 0$ for rotation angle is not 0 or $\pi$). Without losing generality, we have $$Av_1=1v_1$$ $$Av_2=e^{i\theta}v_2$$ $$Av_3=e^{-i\theta}v_3$$
Since $A$ is normal, $A$ and $A^t$ share the same eigenvectors. By the same token, the complex conjugate of the eigenvalue of $A$ is the eigenvalue of $A^t$ corresponding to the same eigenvector, so we have $$A^tv_1=1v_1$$ $$A^tv_2=e^{-i\theta}v_2$$ $$A^tv_3=e^{i\theta}v_3$$
Substituting the above equations into $M=A-A^t$, we have $$Mv_1=0v_1$$ $$Mv_2=(e^{i\theta}-e^{-i\theta})v_2$$ $$Mv_3=(e^{-i\theta}-e^{i\theta})v_3$$.
Then the conclusions are obvious.
Hint: any rotation in ${\mathbb R}^3$ is a rotation around an axis. This is because the characteristic polynomial of $A$ is a cubic with real coefficients, and has a real root (which is $+1$ rather than $-1$ because of the determinant being $1$).
Since $A$ is orthogonal, we know that $AA^T=I$, that is $A^T=A^{-1}$. This implies that $\det(A)=\pm1$. We are given that $\det(A)=1$.
An orthogonal matrix represents an isometry; that is, $$ |Ax|^2=x^TA^TAx=x^Tx=|x|^2 $$ There are two kinds of isometries in $\mathbb{R}^3$: rotations and reflections. rotations have determinant $1$ (they preserve orientation) and reflections have determinant $-1$ (they reverse orientation). Thus, $A$ is a rotation.
Rotations by an angle other than $0$ or $\pi$, have one real eigenvector, the axis of rotation, and it has eigenvalue $1$. Let $v$ be the axis of rotation of $A$. That is, $Av=v$. Multiplying by $A^{-1}$ yields $v=A^{-1}v=A^Tv$. Therefore, $v$ is also an eigenvector of $A^T$ with eigenvalue $1$. Thus, $$ Mv=(A-A^T)v=v-v=0 $$ Suppose that $Mu=0$, then $Au=A^Tu$. Multiplying by $A$ yields $A^2u=u$. Thus, $$ Mu=0\implies(A+I)(A-I)u=0 $$ Since the only real eigenvalue of $A$ is $1$, $A+I$ is invertible, which implies that $(A-I)u=0$. That is, $u$ is parallel to the axis of rotation. Thus, the nullity of $M$ is $1$ and its rank is $3-1=2$.
To answer the second part of the question, we first notice that we are looking for an eigenvector with eigenvalue $1$. That is a $v$ so that $$ \begin{bmatrix} a_{11}-1&a_{12}&a_{13}\\ a_{21}&a_{22}-1&a_{23}\\ a_{31}&a_{32}&a_{33}-1\\ \end{bmatrix} v=0 $$ To get a vector perpendicular to the first two rows of $A$, we can take their cross product: $$ v=\begin{bmatrix} a_{12}a_{23}-(a_{22}-1)a_{13}\\ a_{13}a_{21}-(a_{11}-1)a_{23}\\ (a_{11}-1)(a_{22}-1)-a_{21}a_{12} \end{bmatrix} $$ Since the rows of $A-I$ are linearly dependent, this $v$ is perpendicular to the third row as well. If the previous vector is $0$ (e.g. if the first two rows of $A-I$ are parallel), then try the cross product of the second and third rows of $A-I$: $$ v=\begin{bmatrix} (a_{22}-1)(a_{33}-1)-a_{32}a_{23}\\ a_{23}a_{31}-a_{21}(a_{33}-1)\\ a_{21}a_{32}-a_{31}(a_{22}-1) \end{bmatrix} $$ or the cross product of the third and first rows of $A-I$: $$ v=\begin{bmatrix} a_{32}a_{13}-a_{12}(a_{33}-1)\\ (a_{33}-1)(a_{11}-1)-a_{31}a_{13}\\ a_{31}a_{12}-(a_{11}-1)a_{32} \end{bmatrix} $$
