You have an example of something called a local ring, this is a ring with a unique maximal ideal. You are taking the ring $\mathbb{Z}$ and you are localizing at $(p)$. (In short, localization a ring $R$ at a prime ideal $I$ is just inverting everything that is outside of $I$. The fact that $I$ is prime tells us that $R\backslash I$ is multiplicatively closed, we write this $R_I$).
It is well known that in a local ring $R_I$ the unique maximal ideal is the ideal generated by $I$ in its localization, so in your case the unique maximal ideal will be generated by $(p)$, everything of the form $p\cdot\frac{a}{b}\in R$ (fractions whose numerator is a multiple of $p$).
You could finish your problem using the following lemma:
Lemma: If $R$ is a local ring with unique maximal ideal $I$, then $R\backslash I=R^\times$.
Proof: Let $x$ be a unit. Then $x\notin I$ because the ideal generated by $x$ is $R$ and $I$ is maximal (in particular proper). Hence $x\in R\backslash I$.
Now let $x\in R\backslash I$, consider the ideal $(x)$. Assume for sake of contradiction that $x$ is not a unit, meaning that $(x)$ is proper. Hence, $(x)$ is contained in a maximal ideal, but there is only one such an ideal, so we get $(x)\subset I$, so $x\in I$, a contradiction with the fact that $x\in R\backslash I$, so we get that $(x)$ is not proper, meaning that $x$ is a unit.
Using the above lemma you could finish your problem. You already found all the units. Hence you found $R\backslash I=R^\times=\{\frac{a}{b}: p\nmid a\}$, taking complements: $I=\{\frac{a}{b}:p\mid a\}$.
Here is a reference if you want to see why localization at a prime ideal yields a local ring.