Problem : Suppose that $f$ is continuous on $\mathbb{R}$. Show that $f$ and $\hat f$ cannot both be compactly supported unless $f=0$.
Hint : Assume $f$ is supported in [0,1/2]. Expand $f$ in a Fourier series in the interval [-,1], and note that as a result, f is a trigonometric polynomial.
I proved that f is trigonometric polynomial by using hint. But, I don't know how to prove function's fourier transform cannot compactly supported function. Can I get some hints?
Suppose the support of $f$ is contained in $[-1,1],$ and $\hat f (y) = 0$ for $|y|>N \in \mathbb N.$ Applying a standard Fourier series argument on $[-\pi,\pi]$ then shows
$$f(x) = \sum_{-N}^{N}\hat f (n) e^{inx}, x \in [-\pi,\pi].$$
Thus $f$ is a trigonometric polynomial that vanishes on $[1,\pi].$ But a trigonometric polynomial on $[-\pi,\pi]$ that vanishes at an infinite number of points must vanish identically. Thus $f\equiv 0.$
If $f$ is compactly supported, then $\hat{f}$ is holomorphic. You can see that because the Fourier transform integral extends into the complex plane and is differentiation in the complex variable $s$: $$ \hat{f}(s) = \frac{1}{\sqrt{2\pi}}\int_{-R}^{R}e^{-ist}f(t)dt \\ \hat{f}'(s) = \frac{1}{\sqrt{2\pi}}\int_{-R}^{R}(-it)e^{-ist}f(t)dt. $$ Holomorphic functions cannot have too many zeros, which would happen if $\hat{f}$ were compactly supported.
By the Schwartz's Paley-Wiener theorem the Fourier transform of a compact-supported function is an entire function, and non-zero entire functions cannot be compact supported by Liouville's theorem.
