Firstly there's a small error in your original question when you say
to show the continuity of a function $f : \mathbb{R}^k \to M$, we would consider open sets of $M$, as open sets of $M$, i.e not $\mathbb{R}^n$.
Technically we are considering the open sets of $M$ but
since $M \subseteq \mathbb{R}^n$ and $M$ has the subspace topology inherited from $\mathbb{R}^n$, so the topology of $\mathbb{R}^n$ plays a vital role in determining continuity of $f$.
If I understand your question correctly, there are two parts to it. The first part is can we only rely on the topology of a set to define differentiability? In other words can we define differentiability for maps between arbitrary topological spaces $X$ and $Y$? And the answer to that is no. (see mathcounterexamples.net's answer)
The second part I think has a simple answer. Some authors (expecially for texts on multivariable calculus) only define differentiability for functions $f : U \to \mathbb{R}^n$ where $U$ is an open susbet of $\mathbb{R}^k$. Now I think from this your misunderstanding is that you're not sure whether to take the interior with respect to the subspace topology or the larger topology.
For example you can extend the definition above to an arbitrary set $S \subseteq \mathbb{R}^k$ and then define differentiability of a function $f : S \to \mathbb{R}^n$ on $$\operatorname{Int}_{\mathbb{R}^k}(S)$$ which by definition of the interior is the largest open set in $\mathbb{R}^k$ contained in $S$ (as opposed to $\operatorname{Int}_{S}(S) = S$ which is the largest open set in $S$ with the subspace topology contained in $S$ which is just $S$). If I recall correctly this is what Munkres did in his book.
As a quick application to the example you gave above. By the above definition, the function $f : (0, 1] \to \mathbb{R}^m$ would only be differentiable on $$\operatorname{Int}_{\mathbb{R}}\left((0, 1]\right) = (0, 1)$$
Why do we only define differentiability for open sets in $\mathbb{R}^n$?
To answer this question you have to go back to the definition of a limit of a function (because differentiability is defined through limits of functions)
Definition: [Limit of a function] Let $(X, d_X)$ and $(Y, d_Y)$ be metric spaces. Suppose that $E \subseteq X$. Let $f : E \to Y$ and suppose that $p$ is a limit point of $E$. We say that $$\lim_{x \to p} f(x) = q$$ if there exists a point $q \in Y$ with the property that for all $\epsilon > 0$ there exists a $\delta > 0$ such that $$0 < d_X(x, p) < \delta \implies d_Y((f(x), q) < \epsilon$$ for any $x \in B_{(X, d_x)}(p, \delta)$
Now look at the very last condition there:
for any $x \in B_{(X, d_x)}(p, \delta)$
What this actually states is that there must exist a $\delta > 0$ such that $B_{(X, d_x)}(p, \delta) \subseteq E$, because if $B_{(X, d_x)}(p, \delta) \not\subseteq E$, then there'd exist an $x \in B_{(X, d_x)}(p, \delta)$ such that $x \not \in E$, but then $f$ is not defined at $x$, since it lies outside of $E$, so the statement $$0 < d_X(x, p) < \delta \implies d_Y((f(x), q) < \epsilon$$ is meaningless because $f(x)$ is not even defined.
Thus since there must exist a $\delta > 0$ such that $B_{(X, d_x)}(p, \delta) \subseteq E$, it follows that $p$ is an interior point of $E$. So $p \in \operatorname{Int}_X(E)$ (which is an open set most certainly), that is $p$ is an element of the interior of $E$ with respect to the metric space $X$. Why is this the case? This is because above we've taken the open ball (open set) with respect to the metric space $X$.
That's all fine and good, but how does it affect differentiability you ask?
Well differentiability depends on the limit of a function existing, we've seen above that the limit of a function doesn't exist (or make sense really) if we take the limit of a function at a non-interior point, so we can only take the limits of functions at interior points, which implies that we can only differentiate functions or talk about differentiation of functions defined on open sets. That is the main reason why we only define differentiability for open sets in $\mathbb{R}^n$. (Note how $\operatorname{Int}_X(E)$, translates to $\operatorname{Int}_{\mathbb{R}}\left((0, 1]\right)$ in my example above)
Consider the simple case of differentiability of a function between a subset of $A \subseteq \mathbb{R}$ and $\mathbb{R}$. Consider a function $f : A \to \mathbb{R}$, we define the derivative of $f$ at a point $a \in A$, as $$f'(a) = \lim_{t \to 0} \frac{f(a+t)-f(a)}{t}$$ provided the limit exists. Now for the time being suppose we don't include the added condition that $A$ must contain a neighborhood of $a$. Then note that there is no need for $f(a+t)$ to even be defined, so the limit doesn't make sense and we can't talk about the derivative of the function at $a$. So we really need the condition that $A$ must contain a neighborhood of $a$. I've given a more explicit example of this in the comments below.
Extra material
But there is another problem that I've yet to address, which is that the function $$\frac{f(a+t)-f(a)}{t}$$ isn't even given a domain in most textbooks, and you could say that for large $t$ it wouldn't even be defined if the domain of $f$ wasn't the whole of $\mathbb{R}$. The key realization to make is that we only need the function $\frac{f(a+t)-f(a)}{t}$ to be defined locally on a small neighborhood $\Theta_a$ containing $0$, and note very importantly that the neighborhood $\Theta_a$ containing $0$ depends directly on finding a neighborhood $\Omega$ containing $a$. Most textbooks don't stress the importance of this which is why I formalized the following definition to address these issues
My definition of the derivative of a function at a point: Let $U \subseteq \mathbb{R}$ be an open set. Let $f : U \to \mathbb{R}$ be any function. Pick $a \in U$ and choose $r > 0$ such that $B(a, r) \subseteq U$. Let $\Omega_a = B(a, r) = (a-r, a+r)$ and let $\Theta_{r_a} = B(0, r) \setminus \{0\}$ Define $\phi : \Theta_{r_a} \to \mathbb{R}$ by $$\phi(t) = \frac{f(a+t) - f(a)}{t}.$$ Then we define the derivative of $f$ at $a$ as $$f'(a) = \lim_{t \to 0} \phi(t)$$ provided that the limit exists and we say that $f$ is differentiable at $a$.
You can check based on the definitions I've given that this is indeed well-defined, and it is entirely equivalent to the definitions in Munkres' book and also pay attention to how the need for an open set is vital to this definition. But in practice I would never use this definition, this is just a way to check that everything your textbook authors are saying makes sense and is rigorous and that there is a reason that they've made the choices they have for a definition.
