Let me share to you how I think Rudin did his magic in a more logical/principled way.
So what we want is to show that for any $p \in A = \{ p \in Q_+ : 0<p^2 <2\}$ we can always find a larger $q \in A$. The way I thought of it is that we want a $q = p+\epsilon, \epsilon>0$ such that:
$$ 0<p<(p+\epsilon)^2 = p^2 + 2 \epsilon p + \epsilon^2< 2$$
if you make $\epsilon$ the subject its really difficult to come up with a way to have its solution not involving square roots as you might have noticed. I realized that was really annoying and that was the issue. So with that intuition in mind I tried re-writing $p^2 + 2 \epsilon p + \epsilon^2$ so that maybe with some inspiration I could get rid of the term $\epsilon^2$. The first thing I tried was to write it without any squares at all it by factor $\epsilon$ and $p$. This lead to:
$$ p^2 + 2 \epsilon p + \epsilon^2 = p^2 + \epsilon p + \epsilon p+ \epsilon^2= p(p+\epsilon)+\epsilon (p+\epsilon)$$
then I stared at this equation and wondered if there was anything else I knew about the problem so that I could "get rid" of one of the terms in $\epsilon (p+\epsilon)$ with the goal of removing the $\epsilon^2$ and maybe having something in the rationals for sure. Doing that I noticed involved some type of knowledge of $q = p+\epsilon$ or $\epsilon$. I knew $\epsilon>0$ which was not terribly helpful but I noticed that for sure $p+ \epsilon < 2$ (by drawing some diagrams, then I discovered I could "cheat" by using the fact that I knew $q = p+ \epsilon < \sqrt 2 < 2$ to justify it to myself, though you can easily prove it by contradiction if you want a rigorous reason). At this point I noticed that by using the inequality I actually introduced knew knowledge to the problem and not just playing around with the rules of algebra, which was quintessential because otherwise things are just re-writting things in terms of others for hours as it happened to me.
Anyway this lead to realization that I could indeed remove the $\epsilon^2$ as follows:
$$ p(p + \epsilon) + \epsilon(p+\epsilon) < p^2 + p \epsilon + 2 \epsilon $$
hoping that it wouldn't lead to an upper bound that wasn't too high/large, since I introduced the fact $p+\epsilon < 2$ which is an upper bound (lets hope its not too large!). Right now I intuitively though I bet this will work because I indeed can control how small I make $\epsilon$ in order to make $p \epsilon + 2 \epsilon$ sufficiently small. I proceeded by just enforcing that it was indeed smaller than 2:
$$ q = (p+\epsilon)^2 < p^2 + p \epsilon + 2 \epsilon < 2 $$
then I proceeded to make $\epsilon$ the subject to get:
$$ 0 < \epsilon < \frac{2 - p^2}{p+2} = -\frac{p^2 - 2}{p+2}$$
at this point I realized that choosing $\frac{2 - p^2}{p+2}$ would make the upper bound (introduced after using the fact $q =p+\epsilon<2$) equal to 2 but if we applied it to $q = p+\epsilon$ itself then since it was strictly less than the upper bound, that it we would be safe and have $q$ still be less than $2$. That happened to be correct and so I choose that value $\epsilon = -\frac{p^2 - 2}{p+2} $.
Notice however that any rational constant should work to bound $p+\epsilon$ as long as its constant. Though, larger bounds would obviously require smaller $epsilon$'s (which is fine as long they are rationals).