I read from a paper which says "compact semisimple Lie groups contain no nontrivial unipotent elements". I wonder if and why this is true (how is the compactness used here?).
Definition: $g\in G$ being a unipotent element means that $Ad_g:Lie(G) \to Lie(G)$ is a unipotent transformation of Lie algebras.
You don't need semisimplicity, just compactness. The idea is that unipotent elements look like $g = \left[ \begin{array}{cc} 1 & 1 \\ 0 & 1 \end{array} \right]$ so their powers look like $g^n = \left[ \begin{array}{cc} 1 & n \\ 0 & 1 \end{array} \right]$; note that the coefficient in the top right grows unboundedly. What this means is that the collection of powers $\{ 1, g, g^2, \dots \}$ of $g$ isn't contained in any compact subset (in particular, not in any compact subgroup) of $GL_2$. This kind of thing happens more generally for any unipotent matrix.
To be more precise, we'll prove the following.
Claim: Let $X \in GL_d(\mathbb{R})$ be a nontrivial unipotent matrix (not the identity). Then the powers $\{ 1, X, X^2, \dots \}$ are unbounded with respect to any matrix norm, and so are not contained in any compact subset of $GL_d(\mathbb{R})$.
(To conclude just observe that the image of $G$ under the adjoint action is a compact subgroup of $GL(\mathfrak{g})$.)
Proof. Write $X = 1 + N$ where $N$ is nilpotent; say that $N^r \neq 0$ but $N^{r+1} = 0$ (where $r+1 \le d$). Then
$$X^n = (1 + N)^n = \sum_{i=0}^r {n \choose i} N^i.$$
The coefficients ${n \choose i}, 0 \le i \le r$, are polynomials of degree $i$ in $n$, and as $n \to \infty$ the growth of the above sum is dominated by the term whose coefficient has the highest degree, ${n \choose r} N^r$. In particular by the triangle inequality we have
$$|X^n| \ge {n \choose r} \left| N^r \right| - \left| \sum_{i=0}^{r-1} {n \choose i} N^i \right|$$
and this grows unboundedly as $n \to \infty$. $\Box$
To reduce to the case of orthogonal groups: the Killing form of a compact (semi-simple) Lie group is negative-definite, and the Adjoint action of the group preserves it. Under hypotheses that promise that the kernel of Adjoint is finite (for example), unipotent elements cannot map to $1$, and are still unipotent.
A unipotent element $u$ in an orthogonal group $O(n,\mathbb R)$ has trace $n$. Since rows (and/or columns) or an orthogonal matrix have length $1$, the only way to achieve trace $n$ is that the diagonal entries are $1$... and then the off-diagonal entries must be $0$. (Proving that $u=I$, rather than being a proper unipotent element.)
