It' easy to see that the $ f ( x ) = 0 $ and $ f ( x ) = - x $ are both solutions to the functional equation
$$ f \big( x ^ 2 + y ^ 2 f ( x ) \big) = x f ( y ) ^ 2 - f ( x ) ^ 2 \text . \tag 0 \label 0 $$
We show that those are the only solutions. Letting $ x = y = 0 $ in \eqref{0}, we find out that either $ f ( 0 ) = 0 $ or $ f ( 0 ) = - 1 $. But if $ f ( 0 ) = - 1 $, then letting $ x = 0 $ and $ y = 1 $ in \eqref{0}, we have $ f ( - 1 ) = - 1 $, which by letting $ x = y = - 1 $ in \eqref{0}, gives $ f ( 0 ) = - 2 $, and that leads to a contradiction. Thus we must have $ f ( 0 ) = 0 $, and letting $ y = 0 $ in \eqref{0}, we get
$$ f \big( x ^ 2 \big) = - f ( x ) ^ 2 \text . \tag 1 \label 1 $$
In particular, letting $ x = 1 $ in \eqref{1}, we find out that either $ f ( 1 ) = 0 $ or $ f ( 1 ) = - 1 $. If $ f ( 1 ) = 0 $, then letting $ x = 1 $ in \eqref{0}, we get $ f ( y ) = 0 $, which gives one of the solutions mentioned above. From now on, we assume that $ f ( 1 ) = - 1 $. Letting $ x = 1 $ in \eqref{0}, we get
$$ f \big( 1 - y ^ 2 \big) = f ( y ) ^ 2 - 1 \text . \tag 2 \label 2 $$
Substituting $ - y $ for $ y $, we find out that $ f ( - y ) ^ 2 = f ( y ) ^ 2 $. Now, if we let $ y = 1 $ in \eqref{0}, we have $ f \big( x ^ 2 + f ( x ) \big) = x - f ( x ) ^ 2 $, which shows that if $ f ( - x ) = f ( x ) $ then $ x = 0 $. Combining with the previous result, we get
$$ f ( - x ) = - f ( x ) \text . \tag 3 \label 3 $$
Now, \eqref{1}, \eqref{2} and \eqref{3} show that
$$ f \big( y ^ 2 - 1 \big) = - f \big( 1 - y ^ 2 \big) = - f ( y ) ^ 2 + 1 = f \big( y ^ 2 \big) + 1 \text , $$
which means that we have $ f ( x + 1 ) = f ( x ) - 1 $ for $ x \ge - 1 $. Substituting $ - x - 1 $ for $ x $ in the last equation and using \eqref{3}, we get the same equation for $ x \le 0 $, and thus it's true for every $ x $. By induction and \eqref{3}, we get $$ f ( x + n ) = f ( x ) - n \tag 4 \label 4 $$
for every integer $ n $. In particular, $ x = 0 $ gives $ f ( n ) = - n $ for every integer $ n $. Now, we combine \eqref{0}, \eqref{1}, \eqref{3} and \eqref{4} to get
$$ f \big( n y ^ 2 \big) = - f \big( - n y ^ 2 \big) = - n ^ 2 - f \big( n ^ 2 - n y ^ 2 \big) = - n ^ 2 - n f ( y ) ^ 2 + f ( n ) ^ 2 = - n f ( y ) ^ 2 = n f \big( y ^ 2 \big) \text . $$
Together with the previously proven facts, we can generalize this to
$$ f ( n x - m ) = n f ( x ) + m \tag 5 \label 5 $$
for every integers $ m $ and $ n $. Now, \eqref{1} shows that $ f $ takes nonpositive values at nonnegative points. This shows that if for some integers $ m $ and $ n $ with $ n > 0 $ we have $ \frac m n \le x $, then by \eqref{5}, we get $ f ( x ) \le - \frac m n $. By \eqref{3}, using a similar argument we get that if $ \frac m n \ge x $ then $ f ( x ) \ge - \frac m n $. As the set of rational numbers is dense in the set of real numbers, we must have $ f ( x ) = - x $, which is the other solution mentioned above.
Combining both, we can have some strange functions, correct?
– Lambert macuse Oct 26 '19 at 16:54