It can also be proved by taking the complement. This doesn't answer your question as per the last sentence, but shows how you could extend your arguments in the second paragraph.
Let $$L_1=\{0^p| p\text{ is a prime}\}$$
Then, we have the complement of $L$ to be,
$$\bar{L}=L_1\cup\{\epsilon,0\}$$
where $\epsilon$ denotes the empty string.
$L\cup\bar{L}=\Sigma^*$, where $\Sigma=\{0\}$, the alphabet of the language. Also, $L$,$L_1$, and $\{\epsilon,0\}$ are all pairwise disjoint.
Thus,
$$\begin{align}L_1&=\Sigma^* -(L\cup\{\epsilon,0\})\\
&=\Sigma^*\cap \overline{L\cup\{\epsilon,0\}} \end{align}\tag1$$
Suppose $L$ was regular. We know that $\{\epsilon,0\}$ is regular as the number of strings in the language is finite. By the closure under union, we must have $L\cup\{\epsilon,0\}$ to be regular. Again, by closure under complement, we must have $\overline{L\cup\{\epsilon,0\}}$ to be regular. As $\Sigma^*$, the set of all strings is regular, by closure under intersection, $\Sigma^*\cap \overline{L\cup\{\epsilon,0\}}$ must be regular. From $(1)$, this means that $L_1$ must be regular.
Using the pumping lemma, as suggested by you, it can be shown that $L_1$ is not regular, leading to a contradiction. Thus, the assumption that $L$ is regular is incorrect, $L$ is not a regular language.