You are right, the simple argument you are thinking of requires $f$ to be continuously differentiable. But a more advanced theorem of Darboux states that the derivative of any differentiable function satisfies the intermediate value property, even if it is discontinuous, see How discontinuous can a derivative be?
Suppose, by contradiction, that $f(b)>f(x_0)$ at some $b>x_0$ (the case $b<x_0$ is analogous). By the mean value theorem there is $c\in(x_0,b)$ where $f'(c)=\frac{f(b)-f(x_0)}{b-x_0}>0$. On the other hand, there is some small $\varepsilon>0$ such that $f'(x_0+\varepsilon)<0$. Otherwise, the derivative would be strictly positive on an entire right neighborhood of $x_0$, and it would not be a local maximum. By the Darboux theorem, there is then an $x$ between $x_0+\varepsilon$ and $c$ where $f'(x)=0$. But both of them $>x_0$, so $x>x_0$, in contradiction with the uniqueness of $x_0$.