I guessed $f(a)=a^2$ and $f(a)=0$, but have no idea how to get to the solutions in a good way.
Edit: I did what was suggested:
from $a=b=0$
$f(0)=0$
The function is even, because from $b=-a$
$f(2a^2)=f(-2a^2)$.
I guessed $f(a)=a^2$ and $f(a)=0$, but have no idea how to get to the solutions in a good way.
Edit: I did what was suggested:
from $a=b=0$
$f(0)=0$
The function is even, because from $b=-a$
$f(2a^2)=f(-2a^2)$.
Define $g : \Bbb{R} \to \Bbb{R}$ by $g(x) = f(\sqrt{x})$ and $g(-x) = -g(x)$ for $x \geq 0$. ($g$ is well-defined since $f(0) = 0$.) We claim that
Claim. $g$ solves the Cauchy functional equation $$ g(x+y) = g(x) + g(y), \quad x, y \in \Bbb{R} \tag{1}. $$
Proof. Let $x, y \in \Bbb{R}$.
If $x, y \geq 0$. then we can pick $a\geq b\geq 0$ such that $a^2 - b^2 = \sqrt{x}$ and $2ab = \sqrt{y}$. (This becomes transparent if we write $(a, b)$ in polar coordinates.) Then we have \begin{align*} g(x+y) &= f(\sqrt{x+y}) = f(a^2 + b^2) \\ &= f(a^2 - b^2) + f(2ab) = g(x) + g(y). \end{align*}
If $x, y \leq 0$, then we have $|x| = -x$ and $|y| = -y$ and thus $$g(x+y) = -g(|x|+|y|) = -g(|x|) - g(|y|) = g(x) + g(y)$$ by the definition and $\text{(1)}$.
If $x \leq 0$ and $0 \leq |x| \leq y$, then from $g(y) = g(y-|x|) + g(|x|)$, we have $$ g(x) = -g(|x|) = -(g(y) - g(y-|x|)) = g(y+x) - g(y). $$ Rearrange this to get $g(x+y) = g(x) + g(y)$.
If $x \leq 0$ and $0 \leq y \leq |x|$, then from $g(|x|) = g(|x|-y) + g(y)$ we have \begin{align*} g(x) &= -g(|x|) = -(g(|x|-y) + g(y)) \\ &= -g(-x-y) - g(y) = g(x+y) - g(y). \end{align*} Rearrange this to get $g(x+y) = g(x) + g(y)$.
Interchanging the role of $x$ and $y$, the identity $\text{(1)}$ also holds when $y \leq 0 \leq x$.
These cover all the possible sign combinations of $(x, y)$. Therefore $g$ solves $\text{(1)}$. ////
Conversely, for any $g$ solving the Cauchy functional equation, $f(x) = g(x^2)$ solves the problem. So we obtain a 1-1 correspondence between the solution of
$$ f(a^2 + b^2) = f(a^2 - b^2) + f(2ab), \quad a, b \in \Bbb{R} \tag{2} $$
and the solution of the Cauchy functional equation $\text{(1)}$. Now assuming the Axiom of Choice, the equation $\text{(1)}$ has solutions which is not of the form $g(x) = cx$, which means that $\text{(2)}$ also has solutions which is not of the form $f(a) = ca^2$.
If we assume f is twice differentiable, here is another solution:
We know $f(0) = 0$, let us differentiate equation w.r.p to a, $$2a f'(a^2 + b^2) = 2af'(a^2 -b ^2) + 2b f'(2ab)$$ Let a = b, we have $$f'(0) = 0$$ Let us differentiate the above equation to b,
$$4abf''(a^2 + b^2) = -4abf''(a^2-b^2) + 2f'(2ab) + 4abf''(2ab)$$ Let a = b, we have $$-4a^2f''(0) = 2f'(2a^2)$$ let $x = 2a^2$, so we have $$f'(x) = -2xf''(0)$$ use $f(0) = 0$ and integrate the equation, we have $f(x) = -x^2 f''(0)$. so f is either 0 or f is $kx^2$