The problem is the problem 7 of chapter 1.5. (pp. 63) from the bookComplex Analysis with Applications by Asmar et al. We have to show that for $\theta$ not an integer multiple of $\pi$, $\lim_{n\to\infty}\cos(n\theta)$ and $\lim_{n\to\infty}\sin(n\theta)$ do not exist.
As of the time of writing I haven't really managed to start proving the claim, as my only progress so far is that if $\theta \neq k\pi, k \in \mathbb{Z} \Longleftrightarrow \theta = (l + \epsilon)\pi, l \in \mathbb{Z}, \epsilon \in \mathbb{R}\setminus \mathbb{Z}$, so that $\cos(n\theta) = \cos(n(l + \epsilon)\pi) = \cos(nl\pi)\cos(n\epsilon\pi) - \underbrace{\sin(nl\pi)}_{=0}\sin(n\epsilon\pi) = \pm\cos(n\epsilon\pi)$, depending on whether $2|nl$ or not. This would hint that $\cos(n\epsilon\pi)$ remains oscillating with the $\pm$, but I'm not sure how to finish the proof.
One tool that is given before the problem is that for $z\in\mathbb{C}:\lim_{n\to\infty}z^n = 0$ if $|z| < 1$, $1$ if $z = 1$, and the limit does not exist in other cases. I know that $\cos(t) = (\exp(it) + \exp(-it))/2$, but I'm not yet sure how to actually use this in the proof.