Is it reasonable to assure that p-1 and q-1 aren't smooth?

Question

I came across the requirement that, in RSA, $p-1$ and $q-1$ shouldn't be smooth, shouldn't consist of lots of small factors. Therefore my question:

1. How complicated is it to check whether $p-1$ is smooth? (And how is it done?)

2. Knowing the complexity of finding out, is it reasonable to actually check?

   For instance, does OpenSSL perform this check?

I assume that the chance of hitting smooth $p-1$ among $512$-bit primes (for RSA1024) is pretty low to begin with. Is it?

Answer 1

Checking for smoothness can be computationally expensive, depending on the size of the "small" primes (there is no "natural" definition of "small", one has to define an arbitrary limit). Also, it is not really useful.

The need for non-smooth integers comes from the $p-1$ factorization method. Let $n = pq$ be a RSA modulus that we wish to factor. Now suppose that $x$ is an integer which is a multiple of $p-1$, but not $q-1$. For an integer $v$ relatively prime to $n$ (e.g. $v = 2$), this means that $v^x = 1 \mod p$ but $v^x \neq 1 \mod q$. By the Chinese Remainder Theorem, this implies that $w = v^x - 1 \mod n$ is a multiple of $p$ but not of $q$. A simple GCD computation between $w$ and $n$ then reveals $p$, a prime factor of $n$.

So the $p-1$ factorization method is about assuming that $p$ (or $q$) is such that $p-1$ is $B$-smooth for some limit $B$. Then things go thus:

1. Set $w \leftarrow 2$.
2. For all primes $r$ lower than $B$, compute $w \leftarrow w^{r^{z_r}} \mod n$, where $z_r = \lfloor \frac{\log B}{\log r} \rfloor$.
3. Compute the GCD of $w$ and $n$.

One way to see this method is the following: when we compute modulo $n$, by the CRT, we are actually computing modulo $p$ and modulo $q$ simultaneously. We do multiplications, so we work with two multiplicative groups, the non-zero integers modulo $p$ and the non-zero integers modulo $q$. The goal of the attacker is to find a multiple of the order of one of these two groups. The two groups have orders $p-1$ and $q-1$, respectively. If one of these two orders is $B$-smooth, the algorithm above will work with high probability (there are a few pathological cases where this does not work, due to the condition on $z_r$, but success probability is close to $1-(1/B)$).

This explains the prohibition against smooth $p-1$ and $q-1$. Note, though, that this depends on $B$: $B$ represents the effort that the attacker is ready to invest in the computation. A larger $B$ implies a more expensive attack, but also a more expensive test of non-smoothness, because the best known method for testing $B$-smoothness is trying to divide by all primes lower than $B$: this is somewhat faster than running the $p-1$ factorization method, but not by much. So to defeat an attacker ready to invest one month worth of computation with a dozen PC, we have to do ourselves a check which will take several hours one one PC.

Checking for smoothness is not useful because of the Elliptic Curve Method. ECM is an extension of the $p-1$ method. We again compute things modulo $n$, so that we are working, through the CRT, with two groups, one using integers modulo $p$, the other modulo $q$; and we still hope to hit a $x$ which is multiple of the order of one of the groups.

The groups, in ECM, are randomly chosen elliptic curves. The group law (usually denoted with an addition sign, but that's purely conventional) is weird but can be computed reasonably efficiently through a handful of operations modulo $n$. The interesting point is that when working with integers modulo $p$ (a prime), the order of an elliptic curve modulo $p$ is an integer $f = p+1-t$ where $t$ is called the "trace" and Hasse's theorem states that $|t| \leq 2\sqrt{p}$. So the order of a curve modulo $p$ is "close" to $p$, but still lies in a relatively large range surrounding $p$, and it so happens that a randomly chosen elliptic curve will have an order distributed fairly uniformly in that range.

So the ECM method works by selecting a random curve modulo $n$ (which implicitly selects two random sub-curves, one modulo $p$ and one modulo $q$), assuming that the order of one of these curves is $B$-smooth, and running the same algorithm than for the $p-1$ method, except that we use curve point additions instead of multiplications. If the algorithm fails (the final GCD yields 1), then we can start again with a new random curve.

The ECM method works as long as a randomly selected integer in the range around $p$ (of size $4\sqrt{p}$) has a non-negligible probability of being $B$-smooth from some value $B$ such that the "for all primes lower than $B$" step is computationally feasible. In other words, the RSA modulus will be secure only if the probability of a random integer of the size of $p$ (half the target size of the modulus) of being $B$-smooth is low enough.

Thus, due to ECM, we already rely on the non-smoothness of a random integer. It thus makes little sense to spend CPU cycles on checking $p-1$ specifically: even if we carefully avoid $p$ such that $p-1$ is $B$-smooth, this gains nothing about the probability of a random integer of the size of $p$ to be $B$-smooth. We still need that probability to be negligible, which is achieved by using large enough primes.

Probability of smoothness can be estimated with the de Bruijn function. But, given the discussion above, it is more practical to look at factorization records with ECM. Efficiency of ECM largely depends on the size of the smallest prime factor of the attacked integer -- that's the factor which the ECM finds. The largest factor ever found through ECM is close to $1.8*10^{73}$, i.e. a 244-bit integer. We can thus estimate that 244-bit integers have a probability of smoothness which is high enough for the ECM to work, but larger integers do not. For a 1024-bit RSA key, we use 512-bit prime factors, more than twice that record size. So we can say with some confidence that a random 512-bit integer is sufficiently non-smooth with overwhelming probability, thus "secure enough" for RSA. No need to test.

Answer 2

About difficulty of insuring that $p-1$ is not smooth: it is easy to generate $p$ with assurance that $p-1$ is not smooth. The idea is to first choose a big random prime $p_1$ still much smaller than we want $p$ to be, then test prime candidates $p=2 \cdot k \cdot p_1+1$, for consecutive $k$ such that $p$ is of appropriate size. By construction $p-1$ has a big factor $p_1$, thus is not smooth. The number of $p$ we test on average is unchanged compared to standard methods.

This can be extended to generate $p$ with assurance that both $p-1$ and $p+1$ are not smooth. Techniques for this are detailed in FIPS186-3 appendix B.3, e.g. B.3.6. A similar technique was already in ANSI X9.31:1988 section 4.1.2, which AFAIK was the first official standard for RSA key generation.

Moving to rationale for insuring that $p-1$ is not smooth: The recommendation was in the original 1977 RSA article, as a way to guard against the Pollard's $p-1$ factoring algorithm. The recommendation was extended to $p+1$, as a protection against Willam's $p+1$ algorithm. Although sub-exponential factoring algorithms like ECM and CFRAC made the rationale very dubious, the recommendation was kept in ANSI X9.31:1988. Reportedly, the justification was having a simple answer against an hypothetical lawyer arguing that a digital signature should be rejected as proof because no precaution was taken to guard its modulus against well known attacks.

Insuring non-smoothness of $p-1$ and $p+1$ is now technically pointless in most practical situations; see Ronald R. Rivest & Robert D. Silverman: Are 'Strong' Primes Needed for RSA ?. However, that requirement remains in FIPS186-3 of 2009, when $p$ has 512 bits (but not 1024 bits); that is often justification enough to ensure that non-smoothness.

It remains justified to require non-smoothness of $p-1$ (and to a lesser degree $p+1$) when:

• $p$ is small, say 320 or perhaps 384 bits; nowadays that would mostly be credible in the context of Multi-prime RSA, where the modulus $n$ is taken as the product of more than two primes, say three or four, which speeds up private-key operation considerably over traditional RSA;
• and there are many public keys (millions);
• and the goal of an adversary is to factor the modulus of any one of these many public keys, rather than a particular one (e.g. to discredit the system overall, or mint funny money apparently from some Electronic Purse Smart-Card, which are situations where the adversary is not after a particular key).

In this context, an adversary has both a fair chance of success and a sizable advantage in using $p-1$ factoring (and perhaps $p+1$) on each of the public keys, rather than using ECM or GNFS. In support of that, notice that state of the art factoring programs like MPrime/Prime95 and gmp-ecm try $p-1$ factoring (as improved in Peter L. Montgomery and Robert D. Silverman: An FFT extension of the p-1 factoring algorithm) before ECM: this is because initially (for a random target) it uncovers factors with a significantly better odds/effort ratio than ECM (by a factor of 8 asymptotically, according to Ronald R. Rivest & Robert D. Silverman: Are 'Strong' Primes Needed for RSA ?). With enough potential targets, an attacker may be successful by running only that more efficient algorithm. By making all the keys invulnerable to $p-1$ factoring, we deprive the attacker of this advantage.

To illustrate this numerically: suppose we draw 10 million RSA keys, each with modulus product of three random 320-bit primes. Each resulting 960-bit key is safe from current GNFS and (though only barely) ECM factorization efforts. For each of the $3 \cdot 10^7$ 320-bit primes $p$ drawn, odds that the highest prime factor of the 319-bit $(p-1)/2$ is at most $B2 = 2^{319/6} \approx 10^{16}$ and the second-highest prime factor is at most $B1 = 2^{319/9} \approx 5 \cdot 10^{10}$ are about $3.3 \cdot 10^{-8}$ (this is $G(1/9,1/6)$ in Eric Bach and René Peralta: Asymptotic Semismoothness Probabilities). Thus it is likely that one $p$ matches this condition, and correspondingly that one of the 10 million RSA keys is vulnerable to $p-1$ factoring with smoothness bound parameters $B1$ and $B2$ as above. With $p-1$ factoring as implemented in gmp-ecm 6.3 prepackaged by Ubuntu, one key is tested in less than a day, using a single core of an AMD Athlon 64 and 6GB of RAM. The total runtime is like $2 \cdot 10^4$ core.years, which is better than GNFS projections for 960 bits. These parameters are not optimal, gmp-ecm is not tuned to the max, much of the "stage 1" which largely dominates the runtime uses very little RAM and is a good candidate for offloading to vector processors, so I think the effort can be lowered way further. In this particular case $p-1$ factoring seems better than GNFS, and ECM.

In summary: During RSA key generation, it is easy to choose primes factors $p$ of the modulus in a manner ensuring that $p-1$ is not smooth. This has been customary, and is still required by NIST for 1024-bit keys (but not 2048-bit keys) with two factors. This practice remains justifiable in the case of more factors, below some threshold on the size of the primes, when the attacker would be content with factoring one of many keys. Similar considerations apply to $p+1$.

Answer 3

I'll leave the probability to someone else, but to ensure the non-smoothness of $p-1$ most libraries (including OpenSSL) will generate a prime $p'$ such that $2p' + 1$ is also prime, and then use $p = 2p' + 1$ for the RSA product, and likewise for $q$. This ensures that $p-1$ (which always contains 2 as a factor, of course, $p$ being odd) is of maximal non-smoothness: 2 times a prime. Primality testing is not too expensive, and generating primes $p'$ such that $2p' + 1$ is also prime does take longer, the guaranteed non-smoothness, and thus being immune to certain attacks, is though to be worth it; generating keys is a relatively rare operation anyway.