For $\alpha > 0$, let $n =\lfloor \alpha \rfloor + 1$ and $f : (0,\infty)\to\mathbb{R}$ be continuous. Define $$D^\alpha f(x) = \frac{1}{\Gamma(n - \alpha)} \left(\frac{d}{dx}\right)^n \int_0^x \frac{f(t)dt}{(x - t)^{\alpha - n + 1}},$$ provided the RHS exist for $x\in(0,\infty)$.
Question: Which functions satisfy $D^\alpha f(x) = 0$?
The following is a corollary from Theory and Applications of Fractional Differential Equation by Kilbas, Srivastava, and Trujillo. They state it without proof, just by referring as a Corollary of the fact that $D^\alpha x^{\alpha - k} = 0$ for each $1 \leq k \leq \lfloor\alpha\rfloor + 1$
Corollary 2.1 The equality $D^\alpha f(x) = 0$ is valid if, and only if $$f(x) = \sum_{k = 1}^n c_k x^{\alpha - k}.$$
It is clear that, by direct computation, functions of the form $$f(x) = \sum_{k = 1}^n c_k x^{\alpha - k},$$ where $c_k$ are constants, indeed satisfy this equality. To see this, note that for any integer $1\leq k \leq \lfloor \alpha \rfloor + 1$ we have
\begin{align*} D^\alpha x^{\alpha - k} &= \frac{1}{\Gamma(n - \alpha)} \left(\frac{d}{dx}\right)^n \int_0^x \frac{t^{\alpha - k} dt}{(x - t)^{\alpha - n + 1}} \\ &= \frac{1}{\Gamma(n - \alpha)} \left(\frac{d}{dx}\right)^n \int_0^1 (ux)^{\alpha - k} (x - ux)^{n - \alpha - 1}(x\,du)\\ &= \frac{1}{\Gamma(n - \alpha)}\left( \int_0^1 u^{\alpha - k} (1 - u)^{n - \alpha - 1}\, du\right) \left(\frac{d}{dx}\right)^n x^{n - k}\\ &= 0. \end{align*} The conclusion follows from the linearity of $D^\alpha$.
But are these the only functions?
I have tried the following. Suppose $D^\alpha f(x) = 0$, then \begin{align*} \frac{1}{\Gamma(n - \alpha)} \left(\frac{d}{dx}\right)^n \int_0^x \frac{f(t)dt}{(x - t)^{\alpha - n + 1}} &= D^\alpha f(x) \\ \left(\frac{d}{dx}\right)^n \int_0^x \frac{f(t)dt}{(x - t)^{\alpha - n + 1}} &= 0\\ \int_0^x \frac{f(t)dt}{(x - t)^{\alpha - n + 1}} &= \sum_{k = 0}^{n - 1} c_k x^k. \end{align*}
This is where I got stuck. I could not extract the $f$ in the integral. I was about to use the Fundamental Theorem of Calculus but the denominator would be zero so it wouldn't work.
Update
I tried to substitute $u = t/x$ in the last integral and got
$$\int_0^1 f(ux)(1 - u)^{n - \alpha - 1} du = \sum_{k = 1}^n c_k x^{\alpha - k}$$ I don't know how to continue from here. Am I even on the right track? Any help would be greatly appreciated.
Update II I got the following remark in Fractional Differential Equations An Approach via Fractional Derivatives by Bangti Jin which gives the result with some additional assumptions.
In particular, for $\alpha \in (0, 1)$, we have $(x − a)^{\alpha - 1}$ belongs to the kernel of the operator $D^\alpha$ and plays the same role as a constant function for the first-order derivative. Generally, it implies that if $f, g \in L^1(D)$ with $I^{n - \alpha} f, I^{n - \alpha} g \in AC(D)$ and $n − 1 < α \leq n$, then $$D^\alpha f(x) = D^\alpha g(x) \quad\iff\quad f (x) = g(x) + \sum_{k = 1}^n c_kx^{\alpha - k},$$ where $c_j$, $j = 1, 2, . . ., n$, are arbitrary constants.
Here $I^\alpha$ denotes the fractional integral $$\frac{1}{\Gamma (\alpha)}\int_0^x (x - t)^{\alpha - 1} f(t) dt$$