For an extreme example in which this fails, consider the zero matrix $\left(0\right) \in M_n(k)$. Here, the characteristic polynomial $\chi_0(t)=t^k$, corresponding to the fact that $0$ occurs as an eigenvalue of $(0)$ with multiplicity $n$. Obviously, though, $(0)$ is annihilated by many other endomorphisms, e.g. the identity map.
Your proposition happens to be true if and only if the characteristic polynomial of $A$ equals the minimal polynomial of $A$ (the polynomial that truly satisfies the statement of your proposition), and if you replace "is the characteristic polynomial" with "is a multiple of the characteristic polynomial".
Edit: In the comments below, it's pointed out that $\chi_A$ need not be irreducible for this to work. Of course, if $\chi_A$ is irreducible, then this is true (and what I meant, originally.)