Usually this problem is seen with $g$ being some trigonometric function with the specical condition $\displaystyle \int^1_0 g(x) dx = 0 \ ;$ in this case the result is easy to discover. Intuitively, for large enough $n$ the oscillations become extremely rapid and since piece-wise constant functions uniformly approximate continuous functions arbitrarily well, the areas cancel out. So it is natural to prove this special case first:
Suppose $\displaystyle \int^1_0 g(x) dx =0. $ Let $G(x) =\displaystyle
\int^x_0 g(t) dt.$ Since $g$ is $1$-periodic and $\displaystyle
\int^1_0 g(x) dx=0,$ we have that $G$ is bounded on $ \mathbb{R}.$
If $f \in C^1( [0,1],\mathbb{R} )$ then integration by parts gives $$
\int^1_0 f(x) g(nx) dx = \frac{ f(1)G(n) - f(0)G(0) }{n} - \frac{1}{n}
\int^1_0 f'(x) G(nx) dx \to 0$$ as required. As note that if
$\displaystyle \int^1_0 g(x) dx \neq 0 $ then the limit is not $0$
for arbitrary $f\in C[0,1].$
If $f$ is continuous but not nessicarily $C^1$ then by the
Stone-Weierstrass
Theorem
we can find a continuously differentiable $h: [0,1] \to \mathbb{R}$
such that $\displaystyle \int^1_0 |f(x)-h(x)| dx$ is arbitrarily
small. Then since $$ \biggr| \int^1_0 \left( f(x) - h(x) \right)
g(nx) dx \biggr| \leq \sup_{x\in [0,1]} |g(x)| \int^1_0 | f(x)-h(x)|
dx$$ is also arbitrarily small, we have $$\lim_{n\to\infty} \int^1_0
f(x) g(nx) dx = \lim_{n\to \infty} \int^1_0 h(x)g(nx) dx =0.$$
How can one discover the general result? In the language of linear algebra, what we have shown is that $\displaystyle \int^1_0 f(x) g(nx) dx \to 0$ if and only if $g\in \ker L$, where $L:C[0,1]\to C[0,1]$ is the linear operator defined by $L(g) =\displaystyle \int^1_0 g(x) dx.$ So naturally, we want to see what this kernel looks like. We can see that $L^2=L$ so then we find that $ \ker L = \{ g- Lg : g\in C[0,1] \}.$ So the special case is true if we use $g-Lg,$ and rewriting that gives precisely the general result. We could compress this into a more mysterious conclusion:
Now, for arbitrary $ \displaystyle \int^1_0 g(x) dx$, the function $
\hat{g}(x)=g(x) -\displaystyle \int^1_0 g(x) dx $ is a continuous
$1$-periodic function with $\displaystyle \int^1_0 \hat{g}(x) dx=0$
and applying the previously developed result to it gives the full
result.