In general, when dealing with power series in their convergence domain, we're allowed to interchange derivation with summation, integration with summation, ...
Let's prove it anyway :
Denoting $ \exp $ the function $ \sum\limits_{n=0}^{+\infty}{\left(x\mapsto\frac{x^{n}}{n!}\right)} $, if $ \left(x,h\right)\in\mathbb{R}\times\left(-1,1\right) $, then we have : \begin{aligned}\exp{\left(x+h\right)}-\exp{x}&=\sum_{n=0}^{+\infty}{\frac{\left(x+h\right)^{n}-x^{n}}{n!}}\\ &=h\sum_{n=1}^{+\infty}{\frac{1}{n!}\sum_{k=0}^{n-1}{\left(x+h\right)^{n-1-k}x^{k}}}\\ &=h\sum_{n=1}^{+\infty}{\frac{1}{n!}\sum_{k=0}^{n-1}{x^{n-1}}}+h\sum_{n=1}^{+\infty}{\frac{1}{n!}\sum_{k=1}^{n-1}{\left(x^{n-1-k}\left(x+h\right)^{k}-x^{n-1}\right)}}\\ &=h\sum_{n=1}^{+\infty}{\frac{x^{n-1}}{\left(n-1\right)!}}+h^{2}\sum_{n=1}^{+\infty}{\frac{1}{n!}}\sum_{k=1}^{n-1}{\sum_{i=0}^{k-1}{\binom{k}{i+1}x^{n-i}h^{i}}}\\ \exp{\left(x+h\right)}-\exp{x}&=h\exp{x}+\underset{\overset{n\to +\infty}{}}{\mathcal{O}}\left(h^{2}\right)\end{aligned}
Given $ h $ in $ \left(-1,1\right) $, is not difficult to bound $ \sum\limits_{n=1}^{+\infty}{\frac{1}{n!}}\sum\limits_{k=1}^{n-1}{\sum\limits_{i=0}^{k-1}{\binom{k}{i+1}x^{n-i}h^{i}}} $, anyway, to conclude, we have : $$ \exp'=\exp $$.