Related to this question, if we try and narrow it down a bit. One possible constraint to reduce the number of candidate functions that I could think of is to try and minimize some kind of distance to unity.
Example: the real identity function $$f(x)=x$$ has the non-trivial compositional square root $$f^{\circ \frac 1 2}(x)=k-x$$ This is easy to show: $$( f^{\circ \frac 1 2}\circ f^{\circ \frac 1 2})(x) = f^{\circ \frac 1 2}(f^{\circ \frac 1 2}(x)) = k-(k-x) = k-k+x = x = f(x)$$ But a simpler square root is of course $f$ itself, since $f(f(x)) = x$ if $f(x) = x$!
Can we create some distance measure between functions that will make us find some kind of a "simplest" compositional $n$-th root?
A naive example that would work here for our very simple example above is $$d(x,f^{\circ \frac 1 n}) = \sqrt[p]{\int_{-\infty}^\infty \left|f^{\circ \frac 1 n}(x)-x \right|^pdx}$$
But would that make sense in general? If not, what could we use instead?