The problem is to find all the continuous functions $f:\mathbb{R}\to \mathbb{R}$ defined by :$f(f(x)^2+f(y))=xf(x)+y$
I'm trying my best to figure out a way to find the expression of this unknown function by plugging some numbers, but I could not. Thanks in advance for your help.
The answers are $f(x) = \pm x$.
First, plug in $x = 0$ and $y = 0$, to get that $f(f(0)^2 + f(0)) = 0$. Therefore, let $x = f(0)^2 + f(0)$. Then, we get $f(f(y)) = y$. Then, plug in $y = f(0)^2 + f(0)$. We get that $f(0) = f(0)^2 + f(0)$, or that $f(0) = 0$.
Now, let $f(a) =1$. Clearly, $f(1) =a$ as well. Now, plugging $x = a, y = 1$ in to the original equation, we get that $f(a+1) = a+1$. However, plugging in $x = 1, y = 1$ gives us that $f(a^2 + a) = a + 1$, showing that $a^2 + a = a + 1$, which means that $a = \pm 1$.
Case 1: $f(1) = 1$. Let $x = 1$. Then, we get $f(1+f(y)) = 1+y$. Now, let $y = 1$. Then, we get that $f(f(x)^2 + 1) = xf(x)+1$. However, if, in the first equation, we let $y = xf(x)$, we get that $f(1+xf(x)) = 1 + xf(x)$, which means that $f(x)^2 = x(f(x))$ or that $f(x) = x$.
Proceed similarly in the second case.
Assuming $f$ is differentiable:
$$ \begin{align} f(f(x)^2+f(y)) &= xf(x)+y \\ \frac{\partial{}}{\partial x}\left[f(f(x)^2+f(y))\right] &= f'(f(x)^2+f(y))\cdot 2f(x)f'(x) \\ \frac{\partial{}}{\partial x}\left[xf(x)+y\right] &= xf'(x)+f(x) \\ \Rightarrow f'(f(x)^2+f(y)) &= \frac{xf'(x)+f(x)}{2f(x)f'(x)} \tag{1}\\ \frac{\partial{}}{\partial y}\left[f(f(x)^2+f(y))\right] &= f'(f(x)^2+f(y))\cdot f'(y) \\ \frac{\partial{}}{\partial y}\left[xf(x)+y\right] &= 1 \\ \Rightarrow f'(f(x)^2+f(y)) &= \frac{1}{f'(y)} \tag{2} \end{align} $$ From $(1)$ and $(2)$ we can see that $f'(y)$ is independent of $y$. So $f(x) = ax + b$. Solving this for $f(f(x)^2+f(y))=xf(x)+y$, we get $a=\pm 1$ and $b=0$. So $f(x)=\pm x$.
Obs.: we can discard $f(x) = 0$ and $f'(x) = 0$ by inspection, so $(1)$ is well defined.
By plugging in $x=0$, we see that
$$f(f(y)+f(0)^2)=y,$$
so $f$ is surjective (as $y$ can be any real), injective (as we may find $y$ from $f(y)$), and thus bijective. In addition, letting $x$ be so that $f(x)=0$, we see that
$$f(f(y))=y.$$
Now plug in $x=f(t)$ to see
$$f(f(f(t))^2+f(y))=f(t)f(f(t))+y$$
$$f(t^2+f(y))=tf(t)+y=f(f(t)^2+f(y)).$$
Now, as $f$ is injective, we may apply $f^{-1}$ to both sides to see
$$t^2+f(y) = f(t)^2+f(y) \implies f(t)\in\{-t,t\}.$$
It is easily seen that $f(x)=x$ for all real $x$ and $f(x)=-x$ for all real $x$ are solutions; we now show that they are the only ones. Assume for the sake of contradiction that for some real $x\neq 0$, $f(x)=x$, and for some real $y\neq 0$, $f(y)=-y$. Then
$$f(f(x)^2+f(y))=xf(x)+y \implies f(x^2-y)=x^2+y.$$
This means that $x^2-y=x^2+y$ or $y-x^2=y+x^2$, the first of which implies that $y=0$ and the second of which implies that $x=0$, each a contradiction, finishing the proof.
Note that this doesn't use continuity at all -- so these are the only solutions regardless of whether $f$ is required to be continuous.
