The domain and range of your functions should have a linear structure to begin with. Then, to talk about limits, your space should have a topology. Topological linear spaces are suite for this.
In normed spaced (Banach spaces for example) there is a notion of total derivative that generalizes the concept of derivative known in Calculus.
Definition (Fréchet) Suppose $X,Y$ are normed spaces, $U\subset X$ open. A function $F:U\longrightarrow Y$ is called differentiable at $x\in U$ if there is $F'(x)\in\mathcal{L}(X,Y)$ such that
$$
F(x+h)=F(x)+F'(x)h + r(h)
$$
where $r(h)=o(h)$; i.e., $\lim_{h\rightarrow0}\frac{|r(h)|}{\|h\|}=0$.
This notion does not generalize to locally convex spaces.
There is another option of differentiability that focusses on directional derivatives which can be generalize to general linear topological spaces, in articular locally convex spaces.
Definition: Suppose $X$ and $Y$ are locally convex linear spaces, $U\subset X$ open. Let function $F:U\rightarrow Y$. The directional derivative of $F$ at $\mathbf{x}\in U$ in the direction $\mathbf{v}$ is defined as
$$
D_vF(\mathbf{x}):=\lim_{t\rightarrow0}\frac{F(\mathbf{x}+t\mathbf{v})-F(\mathbf{x})}{t}
$$
when the limit exists (the limit is with respect to the topology in $Y$, that is for any neighborhood $U$ of $\mathbf{0}\in Y$, there is $\delta>0$ such that if $0<|t|<\delta$, $\frac{F(\mathbf{x+v})-F(\mathbf{x})}{t}\in D_vF(\mathbf{x})+ U$).
Definition: $F$ is said to be Gâteaux--differentiable at $\mathbf{x}\in U$ there is a map $L_x:X\rightarrow Y$ such that $D_vF(\mathbf{x})=L_x\mathbf{v}$ for all $\mathbf{v}\in X$. $L_x$ is called Gâteaux--derivative of $F$ at $\mathbf{x}$.
If $X$ and $Y$ are Banach spaces, and
$F$ is differentiable at $\mathbf{x}\in U$, then $F$ is Gâteaux--differentiable at $\mathbf{x}$ and
$D_vF(\mathbf{x})=F'(\mathbf{x})\mathbf{v}$ for all $\mathbf{v}\in X$. The converse is not necessarily true, unless some continuity conditions on $x\rightarrow L_x$ are satisfied.
Theorem:
Suppose $X$ and $Y$ are Banach spaces, $U\subset X$ open, and let $F:U\subset X\rightarrow Y$ be Gâteaux--differentiable on a neighborhood $V\subset U$ of a point $\mathbf{x}\in U$. If the Gâteaux derivative $y\mapsto L_y\in \mathcal{L}(X,Y)$ \is continuous at $\mathbf{x}$, then $F$ is (Fréchet) differentiable at $\mathbf{x}$ and $F'(\mathbf{x})=L(\mathbf{x})$.
Many books in nonlinear functional analysis (Klaus Deimling's for example) have more details and application of Gâteaux differentiation.
