The problem with this is you're taking Leibniz's notation too seriously, and not understanding its limitations is leading you to nonsensical notation and calculations. NEVER EVER write nonsense like $\frac{\partial F}{\partial f}$ or $\frac{\partial F}{\partial f(y,\epsilon=0,\eta)}$ or $\frac{\partial F}{\partial 0}$. You're not differentiating $F$ with respect to the function $f$!
Just so we're clear, I suggest you take a look at the following answers of mine to get an understanding of the "right way" of doing things, and the limitations of Leibniz's notation:
I'm sure I've written several other answers, but I'm too lazy to find them now.
Anyway, in case those do not yet answer your question, here goes. First of all, $F$ is a function of 3 variables, i.e $F:\Bbb{R}^3\to\Bbb{R}$. It takes in an input of three numbers, $(x,y,z)$, and gives you a real number $F(x,y,z)$. It can take in an input of another three numbers $(a,\alpha,\mu)$ and give you an output real number $F(a,\alpha,\mu)$ etc. In your case, $f$ itself is a function, so writing $F(f)$
or $F(x,f,g)$ is just nonsense (sure, people write this often, and I also write such things in my own notes, but that's only because we know what we're doing. If you're just starting to learn things, I would STRONGLY suggest you not take any shortcuts with notation).
When talking about the variation, what is happening is that one defines a map $J:V\to \Bbb{R}$ (often called the "action functional"), where $V$ is a certain space of functions (probably $C^1([a,b],\Bbb{R})$ or maybe a certain subset of it where we fix the endpoints or whatever) as
\begin{align}
J(y)&:=\int_a^bF(x,y(x),y'(x))\,dx
\end{align}
Notice that given $x\in \Bbb{R}$, $y(x)$ is a number, and $y'(x)$ is a number (NOT functions), so their combination $(x,y(x),y'(x))$ is a $3$-tuple of numbers, and thus we can feed it into $F$, and the output we get is another number $F(x,y(x),y'(x))$. Next, given a function $y\in V$ and $\eta\in V$, we think of $y$ as the "point" in $V$, and $\eta$ as the "direction of variation" in $V$. We then consider the single variable function $\Phi:\Bbb{R}\to\Bbb{R}$,
\begin{align}
\Phi(\epsilon)&:=J(y+\epsilon\eta)\\
&:=\int_a^bF\bigg(x, (y+\epsilon\eta)(x),(y+\epsilon\eta)'(x)\bigg)\,dx\\
&=\int_a^bF\bigg(x,y(x)+\epsilon\eta(x),y'(x)+\epsilon\eta'(x)\bigg)\,dx
\end{align}
Remember that for the rest of our discussion, $y$ and $\eta$ are fixed.
Now, to apply the chain rule, you may find it more helpful to introduce a new letters for the functions:
- Define $G:\Bbb{R}^2\to\Bbb{R}^3$ as $G(x,\epsilon):=(x,y(x)+\epsilon\eta(x),y'(x)+\epsilon\eta'(x))$. i.e we have the three component functions $G_1(x,\epsilon)=x$, $G_2(x,\epsilon)=y(x)+\epsilon\eta(x)$ and $G_3(x,\epsilon)=y'(x)+\epsilon\eta'(x)$.
- Note that these functions have partial derivatives given by $(\partial_2G_1)(x,\epsilon)=0$, $(\partial_2G_2)(x,\epsilon)=\eta(x)$ and $(\partial_2G_3)(x,\epsilon)=\eta'(x)$.
Then, our integral definition above can be written as
\begin{align}
\Phi(\epsilon)&=\int_a^bF(G(x,\epsilon))\,dx=\int_a^b(F\circ G)(x,\epsilon)\,dx.
\end{align}
Now, using precise notation, Leibniz's integral rule tells us that
\begin{align}
\Phi'(\epsilon)&=\int_a^b[\partial_2(F\circ G)](x,\epsilon)\,dx
\end{align}
i.e we can move the $\frac{d}{d\epsilon}$ into the integral and it becomes a partial derivative. On the right, $[\partial_2(F\circ G)](x,\epsilon)$ means differentiate $F\circ G:\Bbb{R}^2\to\Bbb{R}$ with respect to its second entry, and only AFTER doing the derivative, we evaluate at the point $(x,\epsilon)$. Now, recall the chain rule:
\begin{align}
[\partial_2(F\circ G)](x,\epsilon)&=\sum_{i=1}^3(\partial_iF)(G(x,\epsilon))\cdot
(\partial_2G_i)(x,\epsilon)
\end{align}
It is very important to understand what this is saying. The subscript of $\partial$ tells you with respect to which entry of the function we're differentiating, and here we're also careful to indicate the points where the derivatives are to be evaluated. Note that this form of the chain rule can also be easily obtained from $D(F\circ G)_{(x,\epsilon)}= DF_{G(x,\epsilon)}\cdot DG_{(x,\epsilon)}$ (matrix multiplication of the Jacobian matrices evaluated at the respective points), and then multiplying by the vector $e_2=\begin{pmatrix}0\\1\end{pmatrix}$ (i.e looking at the second column). Therefore, writing it all out in full,
\begin{align}
\Phi'(\epsilon)&=\int_a^b\partial_2(F\circ G)(x,\epsilon)\,dx\\
&=\int_a^b\sum_{i=1}^3(\partial_iF)(G(x,\epsilon))\cdot (\partial_2G_i)(x,\epsilon)\,dx\\
&=\int_a^b\bigg[\partial_1F(G(x,\epsilon))\cdot \underbrace{(\partial_2G_1)(x,\epsilon)}_{=0}+\partial_2F(G(x,\epsilon))\cdot \underbrace{(\partial_2G_2)(x,\epsilon)}_{=\eta(x)}+\partial_3F(G(x,\epsilon))\cdot \underbrace{(\partial_2G_3)(x,\epsilon)}_{=\eta'(x)}\bigg]\,dx\\
&= \int_a^b\bigg[\partial_2F(G(x,\epsilon))\cdot \eta(x)+\partial_3F(G(x,\epsilon))\cdot \eta'(x)\bigg]\,dx
\end{align}
In particular, if you want the derivative at the point $\epsilon=0$, then
\begin{align}
\Phi'(0)&=\int_a^b\bigg[\partial_2F(G(x,0))\cdot \eta(x)+\partial_3F(G(x,0))\cdot \eta'(x)\bigg]\,dx\\
&=\int_a^b\bigg[\partial_2F(x,y(x),y'(x))\cdot \eta(x)+\partial_3F(x,y(x),y'(x))\cdot \eta'(x)\bigg]\,dx\tag{$*$}
\end{align}
To reconcile $(*)$ with the slightly more classical notation, but still being a little careful so as to not overuse the same symbol twice, let us agree to call the arguments of $F$ as $(\alpha,\beta,\gamma)$, so that we have the three partial derivatives $\partial_1F=\frac{\partial F}{\partial \alpha}$, $\partial_2F=\frac{\partial F}{\partial \beta}$ and $\partial_3F=\frac{\partial F}{\partial \gamma}$. Then, $(*)$ can also be written as
\begin{align}
\Phi'(0)&=\int_a^b\bigg[\frac{\partial F}{\partial \beta}\bigg|_{(x,y(x),y'(x))}\cdot \eta(x)+
\frac{\partial F}{\partial \gamma}\bigg|_{(x,y(x),y'(x))}\cdot \eta'(x)\bigg]
\,dx\tag{$**$}
\end{align}
The last version of the notation is to not introduce new letters $(\alpha,\beta,\gamma)$ and to completely abuse notation by reusing the same symbols to stand for two different things, and not even indicate the point of evaluation of the derivatives:
\begin{align}
\Phi'(0)&=\int_a^b\bigg[\frac{\partial F}{\partial y}\eta+\frac{\partial F}{\partial y'}\eta'\bigg]\,dx\tag{$***$}
\end{align}
Anyway, to summarize: $(*)$ is the most precise way of writing things (slightly cumbersome, I agree, but if you want to learn what is going on, it is necessary to see the gory behind-the-scenes details). $(**)$ is also fine, once one realizes that (as is unavoidable with Leibniz's notation) the letters $\alpha,\beta,\gamma$ are superfluous. $(***)$ is the classical and most condensed form, and the equality in $(***)$ really means "it is equal if the reader knows what's going on".
