Let $g : J \to \mathbb R$ be a function defined on an open interval $J$ containing $0$. Then we write $\lim_{h \to 0} g(h) = a$ if for each $\varepsilon > 0$ there exists $\delta > 0$ such that for all $h \in J$ with $0 < \lvert h \rvert < \delta$ one has $\lvert g(h) - a \lvert < \varepsilon$. Note that $h = 0$ is deliberately excluded here.
If $f : \mathbb R \to \mathbb R$ is a function, we write $\lim_{h \to 0} f(x+h) = a$ if the function $f_x(h) = f(x+h)$ has the property $\lim_{h \to 0} f_x(h) = a$. We say that $f$ is continuous in $x$ if $lim_{h \to 0} f(x+h) = \lim_{h \to 0} f_x(h) = f(x)$.
Now consider the function $f^*_x(h) = f(x+h) - f(x-h)$. Your example shows that if $\lim_{h \to 0} f^*_x(h) = 0$, then $f$ is not necessarily continuous in $x$. We cannot even conclude that $\lim_{h \to 0} f(x+h)$ exists. As an example take $f : \mathbb R \to \mathbb R, f(0) = 0$ and $f(x) = \sin(1/\lvert x \rvert)$ for $x \ne 0$. For $x = 0$ we have $f(h) = f(-h)$, i.e. $f^*_0(h) = 0$ for all $h$.