I will start with general properties of integeres with $p$-adic metric and will provide a concrete example of a Cauchy sequence that is not convergent (which is enough to show that $\mathbb{Z}$ is not compact) later.
Lemma 1. Let $(a_n)$ be a sequence of integers. Then the series $\sum_{i=0}^\infty a_ip^i$ (or more precisely the sequence of partial sums) is a Cauchy sequence in the $p$-adic metric.
Proof. Denote by $S_n=\sum_{i=0}^n a_ip^i$ the sequence of partial sums and note that for any $n$ and any $k,t\geq n$ we have that $p^n$ divides $S_k-S_t$. Thus $d(S_k,S_t)\leq p^{-n}$ gets arbitrarily close to $0$. $\Box$
Lemma 2. Assume that $\sum_{i=0}^\infty a_ip^i$ such that $a_i\in\{0,1,\ldots,p-1\}$ converges to $0$ in the $p$-adic metric. Then $a_i=0$ for all $i$.
Proof. By assumption for any $m$ there is $k$ such that $p^m$ divides $\sum_{i=0}^n a_ip^i$ for all $n\geq k$. By taking sufficiently large $n$ we conclude that $p^m$ divides $\sum_{i=0}^{m-1} a_ip^i$. But since $0\leq a_i \leq p-1$ then
$$\sum_{i=0}^{m-1} a_ip^i\leq \sum_{i=0}^{m-1} (p-1)\cdot p^i=$$
$$=(p-1)\cdot \sum_{i=0}^{m-1} p^i=(p-1)\cdot\frac{p^m-1}{p-1}=$$
$$=p^m-1<p^m$$
All in all $0\leq \sum_{i=0}^{m-1} a_ip^i <p^m$ and so $p^m$ cannot divide it unless all $a_i=0$. $\Box$
Corollary 1. Assume that $\sum_{i=0}^\infty a_ip^i=\sum_{i=0}^\infty b_ip^i$ are both convergent to the same limit in the $p$-adic metric, and both have coefficients in $\{0,1,\ldots,p-1\}$. Then $a_i=b_i$ for all $i$.
Proof. Just apply Lemma 2 to the difference $\sum_{i=0}^\infty (a_i-b_i)p^i$. $\Box$
Corollary 2. $\mathbb{Z}$ is not complete with respect to $p$-adic metric. In particular it is not compact.
Proof. Assume that $\mathbb{Z}$ is complete. Consider the following function:
$$F:(\mathbb{Z}/p\mathbb{Z})^\mathbb{N}\to\mathbb{Z}$$
$$F\big((a_n)\big):=\sum_{i=0}^\infty a_ip^i$$
By Lemma 1 each $F\big((a_n)\big)$ is Cauchy and so our $F$ is well defined since we've just assumed that $\mathbb{Z}$ is complete. But Corollary 1 is just another way of saying that $F$ is injective. This cannot happen because of the cardinality difference between $(\mathbb{Z}/p\mathbb{Z})^\mathbb{N}$ and $\mathbb{Z}$. $\Box$
In particular the proof of the Corollary 2 shows that actually most power series of the form $\sum_{i=0}^\infty a_i p^i$ are not convergent, even though all are Cauchy.
A concrete example of such power series is as follows: let $r\in\mathbb{Z}$, $r> 1$ and consider the geometric series $\sum_{i=0}^\infty r^i$. Since $S_n=\sum_{i=0}^n r^i=\frac{1-r^{n+1}}{1-r}$ then the difference $\frac{1}{1-r}-S_n=\frac{r^{n+1}}{1-r}$. Thus if $r=p^m$ for some $m$, then $\sum_{i=0}^\infty r^i$ series converges to $\frac{1}{1-r}$ in $\mathbb{Q}$. In other words
$$\sum_{i=0}^\infty p^{mi}=\frac{1}{1-p^m}$$
for any $m\geq 1$. Note that the sum is not an integer except for $p=2$ and $m=1$ case.
Also note that the same proof of Corollary 2 works if we replace $\mathbb{Z}$ by $\mathbb{Q}$ on the right side of $F$ function. In particular $\mathbb{Q}$ under $p$-adic metric is not complete as well. A concrete example of a Cauchy sequence that is not convergent over $\mathbb{Q}$ is of course much harder, and I couldn't find an example yet (but will update the answer if I find it).
EDIT: As noticed by @Torsten Schoeneberg in comments, given the standard $p$-adic representation $\sum_{i=0}^\infty a_ip^i$ where $0\leq a_i\leq p-1$ it is known that this series converges to a rational (in $p$-adic metric) if and only if $(a_i)$ is eventually periodic (see this). Thus it is easy to construct such sequence which doesn't converge to any rational, e.g. $a_i=1$ if $i$ is a power of $2$ and $a_i=0$ otherwise.
