Clearly Avitus proves nothing at all !
The key is the derivative of $f(t)=M^k$. Indeed $f'(t)=\sum_{i=1}^k M^{i-1}M'M^{k-i}$. If $M'$ and $M$ commute, then $f'(t)=kM'M^{k-1}=kM^{k-1}M'$. $\exp$ is an entire function, then we can derive the associated series term by term.
$(\exp(M))'=\sum_{i=1}^{\infty} (M^i/i!)'=\sum_{i=1}^{\infty}M'M^{i-1}/(i-1)!=\sum_{i=1}^{\infty}M^{i-1}/(i-1)!M'$ and we are done.
About the converse: assume that for every $t$, $(\exp(M))'=M'\exp(M)=\exp(M)M'$. Does this imply $M'M=MM'$ ? I don't know.
EDIT 1. Assume that if $\lambda,\mu$ are distinct eigenvalues of $M$, then $\exp(\lambda)\not=\exp(\mu)$ ; then $B$ and $M$ commute iff $B$ and $\exp(M)$ commute. Thus if $M$ is $2i\pi$ congruence free (that is, for every $\lambda,\mu\in spectrum(M)$, $\lambda-\mu\notin2i\pi\mathbb{Z}\setminus\{0\})$, then $M'M=MM'$. It remains to study the values of $t$ such that $M(t)$ is not $2i\pi$ congruence free. For instance take $M(t)=\begin{pmatrix}t^2&-2\pi\\2\pi&t\end{pmatrix}$. $\exp(M(0))=I_2$ commute with $M'(0)=\begin{pmatrix}0&0\\0&1\end{pmatrix}$ but $M'(0)$ and $M(0)$ don't commute. I think that, in general, there's not a lot of such values of t (here $t=0$); thus if we assume that $M'$ is continuous, then the result seems to be true. Moreover, note that $(\exp(M))'_{t=0}=1/2I_2$, that is the equality $(\exp(M))'=\exp(M)M'$ is not satisfied when $t=0$.
EDIT 2. Let $ad(M):H\in\mathcal{M}_n:H\rightarrow MH-HM$. Then the derivative of $\exp$ is $D(\exp(M))'(H)=\exp(M)\phi(M)H$ where $\phi(M)=\dfrac{1-\exp(-ad(M))}{ad(M)}$. Here $(\exp(M))'=\exp(M)\phi(M)M'=\exp(M)M'$, that implies $\phi(M)(M')=M'$, that is $M'$ is in the eigenspace of $\phi(M)$ associated to the eigenvalue $1$ (assume $M'\not=0$). Let $(\lambda_i)_i$ be the spectrum of $M$. Then the spectrum of $ad(M)$ is $(\mu)_{i,j}=(\lambda_i-\lambda_j)_{i,j}$ and the spectrum of $\phi(M)$ is $(\rho_{i,j})_{i,j}=(\dfrac{1-\exp(-\mu_{i,j})}{\mu_{i,j}})_{i,j}$. Note that $\rho_{i,j}=1$ implies $\mu_{i,j}=0$ that is the case if $i=j$ OR $\mu_{i,j}\not=0$ and $\exp(-\mu_{i,j})=1-\mu_{i,j}$, equation that has solutions in $\mathbb{C}$.
EDIT 3. OK , the solution is given in W. Ma ,B. Shekhtman. Do the chain rules for matrix functions hold without commutativity ? Linear and multilinear algebra.Vol 58, 1,79-87,2010.
$(A\exp(A)-\exp(A)A)'=0=A'\exp(A)+AA'\exp(A)-\exp(A)A'A-\exp(A)A'=[A,A']\exp(A)$ implies that $[A,A']=0$
Moreover $(\exp(A))'=A'\exp(A)$ does not imply $[A,A']=0$. Take $A=\begin{pmatrix}c&F(t)\\0&0\end{pmatrix}$ where $c\not=0,\exp(c)=1+c$. Note that $A'\exp(A)\not=\exp(A)A'$. The previous counter-example is in $M_2(\mathbb{C})$ ; yet one can deduce a counter-example in $M_2(\mathbb{R})$.