Replacing $A$ by $A^d$ for a maximal proper divisor $d$ of $n$, it suffices to prove the statement in case $n$ is a prime number.
Let us prove that $n\neq p$. By contradiction. Suppose that $n=p$. Write $A=I+p^kB$ with $B\not\equiv0\pmod p$. Then
$$
I=A^p=(I+p^kB)^p=\sum_{i=0}^p\binom{p}{i}p^{ik}B^i
$$
which we want to reduce modulo $p^{k+2}$. Note that the binomial coefficients $\binom{p}{i}$ are divisible by $p$ for $i=1,\ldots,p-1$. Hence
$$
p^{k+2}|\binom{p}{i}p^{ik}
$$
for $i=2,\ldots,p$ since $k\geq1$ and $p\geq3$. It follows that
$$
I\equiv I+p^{k+1}B\pmod{p^{k+2}},
$$
fron which it follows that $B\equiv0\pmod p$. Contradiction.
Therefore, we have $n\neq p$.
Since $A^n=I$, one has $(A-I)(A^{n-1}+\cdots+I)=0$. Since $A\equiv I\pmod p$, one has $A^{n-1}+\cdots+I\equiv nI\bmod p$ is invertible as a matrix with coefficients in $\mathbf F_p$. It follows that $A^{n-1}+\cdots+I$ has a nonzero determinant in $\mathbf Z$. Hence $A-I=0$, that is $A=I$.