Try exploring $$\left(1+\frac 1x\right)^x$$ using the binomial theorem - take $x$ as an integer $n$ to get $$1+n\cdot \frac 1n+\frac {n(n-1)}2\cdot \frac 1{n^2}+\dots=1+1+\frac 12\cdot\left(1-\frac 1n\right)+\dots$$
You can see that the limit is greater than $2$ (if it exists), and with a bit of work you will see that it increases to $e$.
Your original thinking is flawed, possibly because you have noticed that if you keep the exponent the same $$\cfrac {(1+x)^n}{x^n}=\left(1+\cfrac 1x\right)^n$$ the limit is $1$, because you can make the bit inside the bracket as close to $1$ as you like. But that doesn't work unless the exponent is under control.