When you "solve" $f' = f$, you almost get an answer. It's a little like solving $x^2 = 4$. In that case, you get two answers, which are negatives of each other $\pm 2$. And if you have some additional information, like $ x > 0$, then you know your answer is $x = 2$.
For $f' = f$, there's a whole family of solutions, but they have an interesting property: any two of them are proportional. In particular, the function
$$
f(x) = 0
$$
is a solution, but a particularly uninteresting one: it's a multiple of any other solution, but no other solution is a multiple of it. So let's pick some particular nonzero solution of the equation, and call it $g$. It turns out (although this requires proof) that merely knowing that $g(x) \ne 0$ for some $x$ actually means that $g(0) \ne 0$. So at this point we have
$g$ is a solution of $f' = f$.
$g$ is nonzero somewhere, and hence (by a circuitous route) $g(0) = A \ne 0$.
Any other solution $f$ of the equation can be written as $f = cg$, a scalar multiple of $g$.
So (as in the $x^2 = 4$ problem), which value of $c$ do we choose? The answer is that it's arbitrary, but in any field (the real numbers form an algebraic structure called a "field") there's an additive identity $0$, and a multiplicative identity, $1$, and these tend to come up a lot. In choosing the value we want for $g(0)$, we could choose $0$...but that doesn't work because of item 2. So instead, we choose $1$, which means that when we multiply by $g(0)$ in some problem, we'll end up multiplying by $1$, which is easy to simplify.
There's another argument: As it happens, we can prove (with some work involving calculus and the inverse function theorem) that there's a number $e$ with the property that for every integer $n$, we have
$$
g(n) = A e^n
$$
[NB: it's actually true that for any positive rational number $g(n/k) = Ae^{n/k}$, and for irrational numbers $s$, we can then define $Ae^s$ to be the number $g(s)$. In other words: we use this special solution $g$ to define irrational exponentiation.]
Of all possible choices for $A$, the "simplest" one (in our usual notation) is $A = 1$, so that we can write $g(n) = e^n$ (because we can drop any "multiply by 1"). So...that's another good reason to make that choice.