While the analytic definition of a continuous function involves open sets, we can define continuous functions in a completely equivalent way with closed sets. We can define a function between metric (or topological) spaces $f:X \rightarrow Y$ to be continuous if $f^{-1}(C)$ is a closed set if $C$ is.
Typically we avoid closed sets when talking about differentiable functions. Open sets are nice because points are relatively homogenous in them. Locally about any point, the behavior of the set is similar. We can always find an open disk lying in our set about any point. However, on closed sets, we cannot always do this. Any point on the boundary has a different local picture, as any disk containing the point cannot lie solely in this set.
But why is this a problem? Well, in the definition of a real differentiable function of a real variable, we consider the quotient
$$
\lim_{x \rightarrow x_0} \frac{f(x) - f(x_0)}{x-x_0}.
$$
Ideally, if we are working in the set $C$, we would want each $x$ that we are limiting over to be in $C$. Otherwise we would have to refer to the ambient space that $C$ lies in, and it would be a hassle. If $C$ is open, this is fine due to the discussion above.
However, if $C$ is not open, it may be rather strange. What if $x_0$ is an isolated point of $C$ for example? I.e. there is no sequence of points in $C - \{x_0\}$ that approach $x_0$? How do we define the limit here? We cannot simply plug in $x_0$ into the difference quotient as it is not defined there. All in all, this makes it tedious and difficult to talk about differentiability on such sets. Instead, we can just ignore all of these pathologies by concerning ourselves with only open sets.