There are many, many different ways you can approach this.
(1) Define $e^x = \sum_{k=0}^\infty \frac{x^k}{k!}$. Then taking a derivative and passing the limit through (this needs to be justified but it can be done by showing uniform convergence) we can show that $\frac{d}{dx}e^x = e^x$. Alternatively, we can use the limit definition of a derivative and this definition to show this.
(2) Define $e^x = \lim_{n \to \infty} \left( 1 + \frac{x}{n} \right)^n$. See Rene's answer for finding the bound $$\limsup_{h \to 0} \frac{e^h - 1}{h} \leq \frac{k+1}{k}$$ to see how we can compute the limit and hence the derivative.
(3) Define $e^x$ to be the number $y(x)$ such that $\int_1^{y(x)} \frac{1}{t} dt = x$. Use the fundamental theorem of calculus and the chain rule when taking a derivative of the above with respective to $x$ to get
$$\frac{1}{y(x)} y'(x) = 1$$
and conclude $y'(x) = y(x)$ or $(e^x)' = e^x$.
Now all these definitions are equivalent. That is, starting with one definition, we can show that the other must hold. It is clear that (2) and (3) imply (1) because knowing the derivative of $e^x$ you can find it's taylor series. But this then means that (2) and (3) give the same function. So they are all equivalent.
The truly circular way to compute the limit of the finite difference is to use L'Hospital's rule:
$$\lim_{h \to 0} \frac{e^h - 1}{h} = \lim_{h \to 0} \frac{e^h}{1} = 1.$$
The problem is that you use what the derivative of $e^h$ is to show what the derivative of $e^x$ is.