Problem 1. We show that no such $f$ exists.
It is clear that $f$ is surjective and $f'$ is injective. Since $f'$ is continuous, $f'$ is strictly monotone.
Now we show that $f'$ is surjective. We know that $f'(\mathbb{R}) = (\alpha, \beta)$ for some $-\infty \leq \alpha < \beta \leq +\infty$. Then we have to prove that $\alpha = -\infty$ and $\beta = +\infty$. If both $\alpha$ and $\beta$ are finite, then $(f\circ f')(\mathbb{R}) \subseteq f([\alpha, \beta])$, which contradicts the surjectivity of $f$. So it follows that either $\alpha = -\infty$ or $\beta = +\infty$.
- Assume that $f'$ is strictly increasing and $\beta < +\infty$. Then $f'(\mathbb{R}) = (-\infty, \beta)$. This shows that $f$ is strictly decreasing near $-\infty$, and so, $(f\circ f')(\mathbb{R}) = f((-\infty, \beta))$ must be bounded from below, contradicting the functional equation.
Other cases can be treated in a similar way to yield the contradiction, hence we conclude that $f'$ is surjective.
Now since $f'$ is bijective, the functional equation tells that $f$ is also bijective and is the inverse function of $f'$. But this implies that $f$ is monotone, which then contradicts the surjectivity of $f'$.
Problem 2. Again we show that no such $f$ exists.
We make use of the previous problem. To this end, it suffices to show that the given condition implies the continuity of $f'$. The following general lemma will be sufficient for our purpose:
Lemma. Let $g: [a, b] \to \mathbb{R}$ be differentiable and $g'$ be injective. Then $g'$ is continuous.
Proof. By the Darboux theorem, we know that $g'$ possesses intermediate-value property. Together with the injectivity of $g'$, this implies that $g'$ is strictly monotone. In particular, $g'$ can only have jump-discontinuities. Then again by the intermediate-value property, $g'$ cannot have jump-discontinuities. So $g'$ is continuous everywhere.
Problem 3. Again no such function exists.
Assume that $f$ solve the functional equation. Then $f$ is injective and $f'$ is surjective. The former implies that $f$ is strictly monotone, which in turn tells that $f'$ is either non-negative or non-positive. But this contradicts the surjectivity of $f'$.
Addendum. Interestingly, there exists a $C^1$-function $f$ such that $(f' \circ f)(x) = |x|$. Indeed, we can make an ansatz that $f$ takes the form $f(x) = c |x|^{\alpha}$ for some $\alpha > 1$ and $c > 0$ and then plug this to the equation to determine $\alpha$ and $c$. Then we should have $c^{\alpha}\alpha|x|^{\alpha^2 - \alpha} = |x|$, and so, we may choose $\alpha$ as the golden ratio $\phi = \frac{1+\sqrt{5}}{2}$ and then $c = \phi^{-1/\phi}$.
https://mathoverflow.net/questions/337607/existence-of-function-satisfying-ffx-x-almost-everywhere/337611#337611
– Pascal Aug 06 '19 at 18:08