Compare it to the definition of continuity of a function $f$, if $x \in B(a,\delta)$, then $f(x) \in B(f(a),\varepsilon)$. This definition doesn't need $f$ to be defined around a neighbourhood around $a$, in fact $a$ can be an isolated point in the domain of $f$.
(I just checked), when we talk about the (single-var case) derivative of $f$ at $x_0$, we do not need $f$ to be defined in a neighbourhood around $x_0$.
$$ \lim_{x\to x_0} \frac{f(x)-f(x_0)}{x-x_0} = f'(x_0) \iff |x-x_0|< \delta \implies |\frac{f(x)-f(x_0)}{x-x_0} - f'(x_0)| < \varepsilon. $$
We just need that if $B(x_0,\delta)$ contains points, then the inequality above holds.
However, in the $f:\mathbb{R}^p \to \mathbb{R}$ case, $f$ differentiable at $a \iff \exists l(x)$ linear function st. $f(x) = f(a) + l(x-a) + \varepsilon(x)\cdot|x-a|$, where $\varepsilon(x) = 0$ if $x \to a$. Why do we need $f$ to be defined in some neighbourhood here? The equivalent condition for differentiability of $f$ at $a$ is:
$$ \lim_{x\to a} \frac{f(x)-f(a)-l(x-a)}{|x-a|} = 0 \iff \forall x \in B(a,\delta), |\frac{f(x)-f(a)-l(x-a)}{|x-a|}| < \kappa. $$
I don't understand why we need $f$ to be defined in some neighbourhood around $a$.