Rudin needs a rational function $R(x)$ with $R(x)=x$ at $x=\pm \sqrt{2}$, and with rational coefficients, and within that class of functions, one that moves points closer to $+\sqrt{2}$.
The basic idea for finding nice examples, is that $R(x)$ can be taken to be a linear fractional transformation $\frac{ax+b}{cx+d}$. There are theoretical reasons, that Rudin was surely well aware of, why reasonable solutions exist in that class of functions. Given that idea, it's a few lines of simple algebra to identify all possible solutions in that class, and pick out the simplest one.
Why should a linear-fractional solution exist? Ignoring the requirement about the parameters $a,b,c,d$ being rational, the fixed-point conditions are a system of 2 homogeneous linear equations in 4 variables. Solutions therefore exist, as is also clear from projective geometry where any 3 points of the line can be mapped to any 3 other points by a linear-fractional transformation. And the theory of continued fractions, in which $\sqrt{2}$ has a periodic fraction, shows that the rational coefficients requirement can be satisfied in that class of functions; no need to go to degree $2$ or higher. Or take any linear fractional transformation that sends the (unordered) set of two roots to itself, and any rational number $a$ to another rational number $b$. By Galois theoretic symmetry considerations the set of what-maps-to-what is invariant under the conjugation in $\mathbb{Q}(\sqrt{2})$ and hence the function has rational coefficients.
So if you (which is to say, Rudin) knew of those arguments, the first thing to try is linear fractional transformations. Here is the computation:
$a + bX = X(c+dX)$ has solutions $X = \pm \sqrt{2}$.
$dx^2 + (c-b)X - a$ divisible by $(X^2 - 2)$
$a = 2d \quad ; \quad b=c$.
Taking $d=1$ with no lost generality (rescaling of $(a,b,c,d)$ doesn't change the fraction), all such linear fractional function are of the form $\frac{2 + bx}{b + x}$ which is Rudin's function when $b=2$.
To get attraction toward the positive root we need $|f'(\sqrt{2})| \leq 1$. The difference quotient is
$$|\frac{\frac{2+bx}{b+x} - x}{x - \sqrt{2}}| = |\frac{\sqrt{2}+x}{b+x}|$$ so that $|b+x|$ should exceed $|\sqrt{2}+x|$ at $x = \sqrt{2}$. That means $b \geq \sqrt{2}$ which excludes the smallest integer value $b=1$, but $b=2$ works and this is the parameter in the exercise.
The conclusion is that Rudin selected the smallest working example with integer parameters. Because his linear fractional function is attracting at the positive square root, and is continuous on the interval between the roots, it is increasing between them, so it also fits the requirement of sending positive numbers to positive numbers. Hence the ability to simplify some aspects of the problem (the complication that the distance to $\sqrt{2}$ will not be decreased by $R(x)$ for all real numbers) by restricting to positive real numbers.
That is one explanation of how Rudin engineered the problem.