$X$ comprises all and only the elements of $G$ of order $p$, plus the identity. Such elements are grouped in $m$ trivially intersecting subgroups each of order $p$. Therefore:
\begin{alignat}{1}
|X| &= m(p-1)+1 \\
&=mp-(m-1) \\
\tag1
\end{alignat}
But $m\equiv 1\pmod p$, because the number $n_k$ of subgroups of $G$ of order $p^k$ is congruent to $1$ modulo $p$ for every $k=0,1,\dots,k_{\text{max}}$ (see e.g. here), and in particular for $k=1$. Therefore, $m=n_1\equiv 1\pmod p$, and hence from $(1)$ $p\mid |X|$.