I don't think it's necessarily easy to go from what you're doing to what he does.
You are basically looking for some small, positive number $x$ that satisfies the inequality
$$x(2p + x) < 2 - p^2.$$
At this point, there are many ways to pick $x$. For example, you could pick $x$ to be any number that is both less than $p$ and less than $(2-p^2)/3p$.
But if you want to get Rudin's result, you first impose the condition $x < 2 - p$. Now it is sufficient for the inequality
$$x(p+2) \leq 2 - p^2$$ to be satisfied.
So it's enough to take
$$x = \frac{2-p^2}{p + 2}$$
and to check that this value of $x$ satisfies $x < 2-p$. This is straightforward to check.
What I think Rudin is really doing is this. Draw the parabola $y = x^2 - 2$. You already know a point $(p,p^2 - 2)$ on the parabola, and you want to find a better approximation than $p$ to $\sqrt{2}$, the new approximation being on the same side of $\sqrt{2}$ as $p$ is.
To do this you draw a line of positive (rational) slope $m$ through the point $(p,p^2 - 2)$ you know, and take the intersection $(q,0)$ with the $x$-axis. The fact that $m > 0$ guarantees that $q$ and $\sqrt{2}$ are on the same side of $p$. To be certain that you don't overshoot $\sqrt{2}$, all you need to do is make sure that the slope $m$ of the line is larger than that of the secant line from $(p,p^2 - 2)$ to $(\sqrt{2},0)$, which is $p + \sqrt{2}$. For convenience, Rudin takes $m = p + 2$, but he could have taken $p + 3/2$ or a lot of other things.
Addendum
When you reach $x(2p + x) < 2 - p^2$, you have a lot of freedom in how to pick $x$. In this problem, all we need to do is find one positive value of $x$ that works. The condition is equivalent to
$$x < \frac{2-p^2}{2p + x}.$$
Because $x$ appears on the right side, it wouldn't make sense to simply say "Pick $x$ to be less than $(2-p^2)/(2p + x)$." Nonetheless, since $x$ can be picked as small as we like (clearly, if $(p + x)^2 < 2$, this will remain true if $x$ is replaced with a smaller positive number), we can simplify things by assuming that $x$ is less than some convenient value. In particular, $2p + x$ will be much simpler if we replace $x$ with $p$, since $x$ will no longer appear in the expression.
Consider the following chain of inequalities, which may or may not be true for a given value of $x$.
$$x(2p + x) < x(3p) \leq 2 - p^2.$$
The first inequality will be true if $x < p$. The second will be true if $x \leq (2-p^2)/3p$. So any choice of $x$ that is less than the minimum of $p$ and $(2 - p^2)/3p$ will necessarily be a solution of the original inequality.
Addendum 2
In response to a comment, I will add details about the part of the answer that explains Rudin's expression for $q$.
This is very much like what I explained above for $3p$, but with $p + 2$ instead. Essentially, in order for the inequality $x(2p + x) < 2 - p^2$ to hold, it is sufficient for the following two inequalities to hold:
$$x(2p + x) < x(p + 2) \leq 2 - p^2.$$
I have not written $p + 2$ here for any reason other than that it's the particular expression Rudin chose. Other expressions would also work, including $p + 3/2$, or more generally $p + a$ where $a$ is any rational number larger than $\sqrt{2}$.
When does the double inequality hold? The first part holds when $2p + x < p + 2$ or, equivalently, when $x < 2 - p$. The second part holds when $x \leq (2 - p^2)/(p + 2)$. As I mentioned in my answer, the second inequality actually implies the first, because we certainly have $(2-p^2)/(p + 2) < 2 - p$. Therefore it is enough to take $x = (2 - p^2)/(p + 2)$, or any smaller rational number.
Can someone help me show that Rudin's formula for $q$ works?
How was Rudin motivated to come up with his formula for $q$? – CopyPasteIt Jun 14 '17 at 13:06