If $p$ is even, make $a_n^p=1/n$, and $a_n$ alternating, that is, let $a_n=(-1)^n/n^{1/p}$.
If $p$ is odd, this is more interesting, because we cannot argue in terms of conditional vs. absolute convergence. This case has been asked before on this site: See here, where I state a very general theorem (with references) and give details for the case $p=3$, that you should be able to adapt easily to cover any odd $p$. The very general result is that for any set $C$ of positive integers there is a sequence $a_0,a_1,a_2,\dots$ of reals such that for any $k$,
$$ \sum_n {a_n}^{2k-1} $$
converges iff $k\in C$.