Given two functions $f$ and $g$, is there a formula for the Fourier transform of $f \circ g$ in terms of the Fourier transforms of $f$ and $g$ individually?
I know you can do this for the sum, the product and the convolution of two functions. But I haven't seen a formula for the composition of two functions.
Have a look at Bergner et al. 2006 "A Spectral Analysis of Function Composition and Its Implications for Sampling in Direct Volume Visualization" IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 12, NO. 5. Sec. 3.
Unfortunately, it's not quite as simple as the transform of a convolution or even the derivative. The short answer is, in one dimension let \begin{equation} P(k,l) = \int_{x \in \mathbb{R}} e^{i2\pi(lg(x) - kx)}\,dx. \end{equation} The FT of the composite function is \begin{equation} \text{FT}\left[f\left(g(x)\right)\right](k) = \int_{l\in\mathbb{R}} \hat{f}(l)P(k,l) \,dl, \end{equation} where $\hat{f}(l)$ is the Fourier transform of $f(x)$. As you can see the transformation involves the inner product of $\hat{f}(l)$ with a slightly awkward two-dimensional function. In the discrete case, this would be implemented as a matrix multiplication.
There is no such rule in general. The key here is variable substitution: If $g$ is a bijection and smooth enough then, if all integrals exist: $$ (\widehat{f\circ g})(\xi) = \int f(g(x))\exp(ix\xi)dx = \int f(y)\exp(ig^{-1}(y)\xi)|\det g'(y)|^{-1}dy.$$ This does only rarely lead to something interesting, e.g. in the case of scaling (i.e. linear transformation of the variable): Working in $\mathbb{R}^d$ with $A\in\mathbb{R}^{d\times d}$ invertible: $$ (\widehat{f\circ A})(\xi) = |\det A^{-1}|\widehat{f}(A^{-T}\xi). $$
Preamble: I thought I would write out what's going in Lee's answer in detail. The paper he cites gives a derivation that is shorter, but somewhat more mysterious to me than the approach below.
I'll give that derivation that at the very bottom -- so if you just want a quick proof go there.
For simplicity of notation*, we will work in an Abelian group $G$, and later on we'll plug in $\mathbb{Z}_n^m$. I'll do this in the case of a finite group so that we can avoid issues of integrability.
(*Some may disagree, but for me a proliferation of Greek letters is worth the clarity about objects.)
High level intuition: The formula we are deriving follows from the following series of observations:
First, that composition can be represented by a composition operator $R_T$, mapping $f \to f \circ T$, and that this is a linear operator. The conjugation of this operator by the Fourier transform $F$ gives an operator $\hat{R_T}$ that tells us how to calculate $\hat{ f \circ T}$ from $\hat{f}$: we define $\hat{R_T}$ to have $F ({ f \circ T}) = \hat{R_T}(\hat{f})$. Since $\hat{R_T}$ is linear, we can check its behavior on a simple basis in order to write down a matrix describing it, and we obtain the formula from Lee's answer by choosing the basis of indicator functions. We get the formula for the matrix entries by an application of Parseval's theorem.
Formal Derivation:
Notation: Let $\hat{G}$ denote the group of characters of $G$, $\hat{G} = \{ \text{group homomorphisms } \chi : G \to \mathbb{C}^* \}$. Additionally, let $F : L^2(G) \to L^2(\hat{G})$ denote the Fourier transform: $F(f)(\chi) = \sum_{g \in G} f(g) \overline{\chi}(g)$ so $f = \frac{1}{|G|} \sum_{\chi \in \hat{G}} F(f)(\chi) \chi$. By basic Fourier analysis, $(F^{-1}(h))(g) = \frac{1}{|G|} \sum_{ \chi \in \hat{G} } h( \chi) \chi(g)$. We'll also write $F(f) = \hat{f}$.
Suppose we have a function $T : G \to G$ -- we assume no structure about $T$, it is just some transformation. Let $R_T : L^2(G) \to L^2(G)$ be given by $R_T(f) = f \circ T$.
The crucial observation is that $R_T$ is a linear function. We let $\hat{R_T} = F \circ R_T \circ F^{-1} : L^2(\hat{G}) \to L^2(\hat{G})$.
$\hat{R_T}$ has the property that $F ( f \circ g) = \hat{R_T} ( \hat{f})$ : the right hand side is $(F \circ R_T \circ F^{-1} \circ F)(f) = F \circ R_T) f = F( f \circ T)$. In particular, it gives the formula for a for the Fourier transform of a composition.
Since $\hat{R_T}$ is a composition of linear functions, it is linear. Thus, to understand $\hat{R_T}$ it suffices to examine its behavior on a favorite basis.
Our favorite basis will be $B = \{ \delta_{\chi} : \chi \in \hat{G} \}$, where $\delta_{\chi}$ is $1$ on $\chi$ and zero elsewhere.
One can compute from the formula that: $F^{-1} (\delta_{\chi}) = \frac{\chi}{|G|}$. Thus, $\hat{R_T} ( \delta_{\chi} ) = F \circ R_T \circ F^{-1} ( \delta_{\chi} ) = F ( \frac{ \chi}{|G|} \circ T )$.
We note Parseval's formula that that $(h, F(f))_{L^2(\hat{G})} = |G| (F^{-1}(h), f)_{L^2(G)}$ , where $(s,t)_{L^2(G)} = \sum_{g \in G} s(g) \overline{t} (g)$ and $(p,q)_{L^2(\hat{G})} = \sum_{\chi \in \hat{G}} p(\chi) \overline{q}(\chi))$. (I believe this is how Parseval's looks in the conventions I chose -- I expanded and verified.)
Additionally, we note that if $v = \sum_{\chi} a_{\chi} \delta_{\chi}$, then $\overline{a_{\chi}} = (\delta_{\chi}, v)$.
Finally, if we want the $\delta_{\Psi}$ coefficient of $\hat{R_T} ( \delta_{\chi} )$, we calculate with Parseval:
$( \delta_{\Psi} , \hat{R_T} ( \delta_{\chi} ) )_{L^2(\hat{G})} = ( \delta_{\Psi} , F ( \frac{ \chi}{|G|} \circ T ) )_{L^2(\hat{G})} = |G| ( F^{-1} (\delta_{\Psi}) , \frac{ \chi}{|G|} \circ T )_{L^2(G)} = |G| ( \frac{\Psi}{|G|}, \frac{ \chi}{|G|} \circ T)_{L^2(G)} = \frac{1}{|G|} ) ( \Psi, \chi \circ T) = \frac{1}{|G|} \sum_{g \in G} \Psi(g) \overline{\chi}(T(g)).$
This gave us the complex conjugate of the $ \delta_{\Psi}$th coefficient, so if we express the linear transformation $\hat{R_T}$ in the basis $B$ , the $(\Psi, \chi)$th entry is $\frac{1}{|G|} \sum_{g \in G} \overline{\Psi(g)} {\chi}(T(g))$. We denote this by $P( \Psi, \chi)$.
In particular, $\hat{R_T}( \hat{f})(\Psi) = \hat{R_T}( \sum_{ \chi \in \hat{G} } \hat{f}(\chi) \delta_{\chi})(\Psi) = \sum_{\chi \in \hat{G} } \hat{f}(\chi) \hat{R_T}( \delta_{\chi} )(\Psi)= \sum_{\chi \in \hat{G} } \hat{f}(\chi) P( \Psi, \chi)$.
Summarizing, we have proved:
If we define $P( \Psi, \chi) := \frac{1}{|G|} \sum_{g \in G} \overline{\Psi}(g) {\chi}(T(g))$, then $F( f \circ T)(\Psi) = \sum_{\chi \in \hat{G} } \hat{f}(\chi) P( \Psi, \chi).$
Specialization to the one dimensional case:
Now we specialize to the case $G = \mathbb{Z}_n$ to obtain the formula. In this case, we identify $\hat{G}$ with $\{\chi_k : k \in \{0,1,2,\ldots, n-1\} \}$, where $\chi_k(x) = \exp(2 \pi i kx)$.
To coordinate with Lee's answer, we will denote the $(k,l)$th entry of $\hat{R}_T$ by $P(k,l)$. Then, $P(k,l) = \frac{1}{n} \sum_{x \in \mathbb{Z}_n} \overline{\chi_k(x)} {\chi_l}(T(x)) = \frac{1}{n} \sum_{x \in \mathbb{Z}_n} \exp ( 2 \pi i (lT(x) - kx)$.
Moreover, we can write $F ( f \circ T) (k) = \hat{R_T}( \hat{f})(k) = \sum_{l \in \{0, \ldots, n-1\}} \hat{f} (l) P(k,l)$.
This is now recognizable as the formula from Lee's answer.
Sanity check 1):
We use the special case $G = \mathbb{Z}_n$ and $T(x) = ax \mod n$ to sanity check the formula. For simplicity let's assume that $a \in \mathbb{Z}_n^*$.
$P(k,l) = \frac{1}{n} \sum_{x \in \mathbb{Z}_n} \exp ( 2 \pi i (lax - kx) = \delta_{la - k}$.
So, $F ( f \circ T) (k) = \sum_{l \in \{0, \ldots, n-1\}} \hat{f} (l) \delta_{la- k} = \hat{f}(ka^{-1})$.
We compute $F ( f \circ T)$ directly: $F( f \circ T)(k) = \sum_{g \in G} f(ag) \overline{ \chi_k(g)} = \sum_{h \in G} f(h) \overline{ \chi_k(a^{-1}g) } = \sum_{h \in G} f(h) \overline{ \chi_{a^{-1} k}(g)} = F(f)(a^{-1} k).$
So that seems good.
Sanity check 2): $G = \mathbb{Z}_n$ and $T(x) = x + a \mod n$.
Then, $P(k,l) = \frac{1}{n} \sum_{x \in \mathbb{Z}_n} \exp ( 2 \pi i (lT(x) - kx) = \frac{1}{n} \sum_{x \in \mathbb{Z}_n} \exp ( 2 \pi i (l(x + a) - kx) = \delta_{k - l} \exp( 2 \pi i la).$
So, $F ( f \circ T) (k) =\sum_{l \in \{0, \ldots, n-1\}} \hat{f} (l) P(k,l) = \sum_{l \in \{0, \ldots, n-1\}} \hat{f} (l) \delta_{k - l} \exp( 2 \pi i la) = \hat{f}(k) \exp( 2 \pi i ka).$
This is a familiar formula, but to be careful let's calculate it directly:
$F( f \circ T)(k) = \sum_{g \in G} f(g + a) \overline{ \chi_k(g)} = \sum_h f(h) \overline{ \chi_k(h - a)} = ( \sum_h f(h) \overline{ \chi_k(h)} ) \overline{ \chi_k( - a)} = F(f)(k) exp( 2 \pi i k a).$
Seems good. :-)
Specialization to the multi dimensional case:
Let's do the case $G = \mathbb{Z}_n^m$.
We identify $\hat{G} = \{\chi_{k,l} : k \in \{0,1,2,\ldots, n-1\}^m \}$, where $\chi_{k}(x) = \exp(2 \pi i k \cdot x)$.
Then:
$P(k,l) = \frac{1}{n^m} \sum_{x \in \mathbb{Z}_n} \overline{\chi_k(x)} {\chi_l}(T(x)) = \frac{1}{n^m} \sum_{x \in \mathbb{Z}_n} \exp ( 2 \pi i ( l \cdot T(x) - k \cdot x)$
$F ( f \circ T) (k) = \sum_{l \in \{0, \ldots, n-1\}^m} \hat{f}(l) P(k,l)$
Comments on complexity:
Computing the matrix $P(k,l)$ is expensive -- naively it takes time $|G|^3$, although perhaps there's some FFT like dynamic programming trick that improves the speed. I think that in situations where you have an efficient implementation of $R_T$, it would be better to compute the operator $\hat{R_T}$ as the composition $F R_T F^{-1}$, where $F$ is implemented using the FFT. You could also compute the matrix $\hat{R_T} = F R_T F^{-1}$ using fast matrix multiplication, if you could write down the matrix of $R_T$ efficiently, I think this would be $O(|G|^{\omega})$, where $\omega$ is matrix multiplication exponent.
Alternative Derivation:
This is along the lines of the derivation in Bergner et al. 2006 "A Spectral Analysis of Function Composition and Its Implications for Sampling in Direct Volume Visualization."
Notation (same as above): We work in an Abelian group $G$. Let $\hat{G}$ denote the group of characters of $G$, $\hat{G} = \{ \text{group homomorphisms } \chi : G \to \mathbb{C}^* \}$. Additionally, let $F : L^2(G) \to L^2(\hat{G})$ denote the Fourier transform: $F(f)(\chi) = \sum_{g \in G} f(g) \overline{\chi}(g)$ so $f = \frac{1}{|G|} \sum_{\chi \in \hat{G}} F(f)(\chi) \chi$. By basic Fourier analysis, $(F^{-1}(h))(g) = \frac{1}{|G|} \sum_{ \chi \in \hat{G} } h( \chi) \chi(g)$. We'll also write $F(f) = \hat{f}$.
The trick is to write $(f \circ T)(x) = (F^{-1} (F (f)))( T(x)) = \frac{1}{|G|} \sum_{\chi \in \hat{G}} \hat{f}(\chi) \chi(T(x))$.
We then use this formula in the calculation of $F( f \circ T)$:
$F( f \circ T) ( \psi) = \sum_{g \in G} ( f \circ T) (g) \overline{\psi}(g) = \sum_{g \in G} \frac{1}{|G|} \sum_{\chi \in \hat{G}} \hat{f}(\chi) \chi( T(g)) \overline{\psi}(g) = \sum_{\chi \in \hat{G} } \hat{f}(\chi) \frac{1}{|G|} \sum_{g \in G} \chi(T(g)) \overline{\psi}(g).$
Letting $P( \psi, \chi) = \frac{1}{|G|} \sum_{g \in G} \chi(T(g)) \overline{\psi}(g)$, the above becomes $F( f \circ T) ( \psi) = \sum_{\chi \in \hat{G} } \hat{f}(\chi) P( \psi, \chi)$.
This proof has the advantage of being shorter and also working for $\mathbb{R}$, since we no longer have to chase the behavior on basis vectors. This also gives us a third sanity check.
On the other hand, it strikes me as very tricky, unlike the more conceptual "follow your nose" type of calculation above.
This is the Lee that gave the answer in 2014, now posting from a registered account. That answer is a simplified version of the multidimensional case. If you want to understand the multidimensional case I suggest a Google search for the paper I referenced.
Best of luck.
All of the other answers detail transforming $f(g(x))$ given $\mathcal{F}f$. This didn't work for me, who had $\mathcal{F}g$ instead, so using the Wikipedia article on Fourier Transforms, I quickly came up with this:
We define $\hat{g}(\nu)$ as the Fourier Transform of $g(x)$, using the convention: $$\mathcal{F}[a(x)](\nu)=\int_{-\infty}^\infty dx a(x) e^{-i x \nu}$$ We also define $\tilde{f}[\hat{g}(\nu)](\nu)$ as the Fourier Transform of $f(g(x))$, $\mathcal{F}[f(g(x))] (\nu)$. Note that $\mathcal{F}[f(g(x))] (\nu)=\tilde{f}[\hat{g}(\nu)](\nu)\neq\hat{f}(\hat{g}(\nu))=\mathcal{F}[f(x)](\mathcal{F}[g(x)](\nu))$.
If $f(x)$ is an analytic function of $x$, we can expand $f(x)$ as a Taylor series: $$f(x)=f(0)+f'(0) x+\frac{1}{2!} f''(0) x^2+\frac{1}{3!} f'''(0) x^3+...$$ Then: $$\tilde{f}[\hat{g}(\nu)](\nu)=2 \pi f(0) \delta(\nu)+f'(0) \hat{g}(\nu)+\frac{1}{2!}f''(0)\int_{-\infty}^{\infty}\frac{d\nu_0}{2 \pi}\hat{g}(\nu_0)\hat{g}(\nu-\nu_0)+\frac{1}{3!}f'''(0)\iint_{-\infty}^{\infty}\frac{d\nu_0}{2 \pi}\frac{d\nu_1}{2\pi}\hat{g}(\nu_0)\hat{g}(\nu_1-\nu_0)\hat{g}(\nu-\nu_1)+...$$ Notice how the multiplication in the Taylor series is transformed into repeated convolution, and how the unit is transformed into the Dirac Delta Distribution.
Starting from Lee's answer, we can write the following:
\begin{equation*} P(k,l) = \int_{x \in \mathcal{R}} e^{i2\pi(lg(x) - kx)}\,dx \end{equation*}
\begin{equation*} P(k,l) = \int_{x \in \mathcal{R}} e^{i2\pi \, l \, g(x)} \, e^{-i2\pi kx} \,dx \end{equation*}
The Jacobi-Anger expansion tells us the following: \begin{equation*} e^{i z \cos \theta} \equiv J_0(z)\, +\, 2\, \sum_{n=1}^{\infty}\, i^n\, J_n(z)\, \cos\, (n \theta) \end{equation*} then \begin{equation*} e^{i z g(x)} \equiv J_0(z)\, +\, 2\, \sum_{n=1}^{\infty}\, i^n\, J_n(z)\, \cos\, (n \arccos(g(x))) \end{equation*}
\begin{equation*} e^{i z g(x)} \equiv J_0(z)\, +\, 2\, \sum_{n=1}^{\infty}\, i^n\, J_n(z)\, T_n \big( g(x) \big) \end{equation*} where $T_n$ is the nth Chebychev polynomial and $J_n$ the nth Bessel function of the first kind.
If we use it in the expansion of $P(k,l)$, we have: \begin{equation*} P(k,l) = \int_{x \in \mathcal{R}} \Big[ J_0(2\pi\,l)\, +\, 2\, \sum_{n=1}^{\infty}\, i^n\, J_n(2\pi\,l)\, T_n \big( g(x) \big) \Big] \, e^{-i2\pi kx} \,dx \end{equation*}
\begin{equation*} P(k,l) = J_0(2\pi\,l)\, \int_{x \in \mathcal{R}} e^{-i2\pi kx} \,dx \, + \, 2\, \sum_{n=1}^{\infty}\, i^n\, J_n(2\pi\,l)\, \int_{x \in \mathcal{R}} T_n \big( g(x) \big) \, e^{-i2\pi kx} \,dx \end{equation*}
\begin{equation*} P(k,l) = J_0(2\pi\,l) . \delta(k) \, + \, 2\, \sum_{n=1}^{\infty}\, i^n\, J_n(2\pi\,l)\, g_n (k) \ \ \ \ with \ \ g_n (k) = \int_{x \in \mathcal{R}} T_n \big( g(x) \big) \, e^{-i2\pi kx} \,dx \end{equation*}
\begin{equation*} \mathscr{F} \Big[ f \big( g(x) \big) \Big](k) = \int_{l \in \mathcal{R}} F(l) \, P(k,l) \, dl \end{equation*}
\begin{equation*} \mathscr{F} \Big[ f \big( g(x) \big) \Big](k) = \delta(k) . \, \int_{l \in \mathcal{R}} F(l) \, J_0(2\pi\,l) \, dl \ \ + \ \ 2\, \sum_{n=1}^{\infty}\, i^n\, g_n (k) \, \int_{l \in \mathcal{R}} F(l) \, J_n(2\pi\,l) \, dl \end{equation*}
And finally we have: \begin{equation*} \boxed{ \begin{split} \mathscr{F} \Big[ f \big( g(x) \big) \Big](k) = & \, f_0 . \delta(k) \ \ + \ \ 2\, \sum_{n=1}^{\infty}\, i^n\, f_n . g_n (k) \\ & \, with \ \ f_n = \int_{l \in \mathcal{R}} F(l) \, J_n(2\pi\,l) \, dl \\ & \, and \ \ g_n (k) = \int_{x \in \mathcal{R}} T_n \big( g(x) \big) \, e^{-i2\pi kx} \,dx \end{split} } \end{equation*}
So, the Fourier transform of the composition seems to have an expression as a Dirac term and a series involving:
I made some sanity checks that seem to be ok but I'm not sure the final formula is bullet proof in every cases.
