Does a high exponent compensate for a low degree of certainty?

Question

If a RSA certificate is created with a low degree of certainty, does the value of the exponent compensate for this?

I'm asking because certain implementations of key generation software hide the certainty value from the administrator, and I'd like to find reassurance in any other parameter.

If the exponent doesn't compensate for this, does a longer key length compensate for certainty?

Answer 1

Well, no, having a longer exponent really doesn't compensate for low degree of certainty in the RSA key generation.

As you are well aware, the most common method for generating RSA keys involves finding primes by picking numbers, and using a probabilistic method to verify whether those numbers are prime. These probabilistic methods have a small probability of claiming that a composite number is prime; the certainty factor that some implementations ask for is an indication of how small this error probability should be.

Now, if the certainty factor is too low, the error that might result is that one of the "primes" that are picked isn't really a prime. This can cause two errors:

• The RSA key might not work (or, at least, not consistently). For example, if the RSA key is used for public key encryption, the private key might not correctly decrypt something encrypted with the public key.

• The RSA key might work; this will happen if the 'prime' that got chosen happens to be a Carmichael number; that is, a composite number $n$ for which $x^n = x \mod n$ holds for all $x$. Now, Carmichael numbers are rare (far rarer than primes), but they do exist. The potential problem is that Carmichael number often have small factors; if someone notices that the RSA key has a small factor, they might be able to figure out the other factors of the Carmichael number, and thus factor the RSA key.

In both cases, using a larger public exponent won't help. A larger RSA key also won't help much for case 1 (other than making it less likely that a nonprime will sneak through). It might for case 2, but I wouldn't be real confident about it.

Now, if the key generation software doesn't allow you to specify a certainty factor, it means that either:

• The software has a preset certainty factor build in; one would hope that it is set fairly conservatively (perhaps about as likely as hardware error), but you can certainly ask

• The software doesn't use a probabilistic method. There are deterministic methods of finding primes (such as the Shawe-Taylor algorithm, see A.1.2.1) that are approximately as efficient as probabilistic methods; perhaps those libraries do that.

On the other hand, if you don't trust some key generation software to do the right thing, perhaps you shouldn't use them. There are some nasty things that a bugged RSA key generation software can do.

Answer 2

No, in RSA, a bigger public exponent $e$ does not compensate in any way for setting a certainty parameter incorrectly. A bigger key length does, by reducing to some degree the odds that a composite is selected (as most pseudo-prime tests, including Miller-Rabin, have odds of missing a composite that lowers with the size of the number tested). But this should seldom be a worry, for several reasons:

• If this incorrect setting of certainty has the consequence of letting a composite creep during the RSA public modulus generation, it is overwhelmingly likely that the resulting private key is non-functional, and in many (but not all) scenario that will get noticed soon, the first time both private and public sides of the key are used together; e.g. a certificate made with this key will almost certainly not check. Many RSA key generation procedures include a check that the key works, perhaps indirectly (e.g. sometime a certification request is self-signed, and that signature checked), and if that's the case, it acts as a double check against inadvertent use of a composite rather than a prime.
• When generating a prime using the BigInteger package of the Java standard library (as in Java\jdk1.7.0_03\src.zip, then java/math/BigInteger.java), even if certainty is set to no more than 1, the function using it, primeToCertainty, will still perform at least one Miller-Rabin test, and one Lucas test (misnamed Lucas-Lehmer, with reference to ANSI X9.80 which had this mistake in the 2001 edition). Assuming these tests are correct, which I did not verify, odds that a composite creeps uncaught are so remote as to be negligible. In these two papers, the Lucas test is studied, a prize is offered for a counterexample with the Miller-Rabin test to base 2, and that's still an open question. Even with certainty set to 0 or negative, the Lucas test remains, and it is extremely unlikely that a 512-bit composite gets uncaught.

Note: I fail to locate the RSA key generators in the Java standard library, and do not know what algorithm they uses, or if certainty has an effect on them.

Update: as pointed in this excellent other answer, if the composite that evaded the primality test happens to be a Carmichael number, the RSA key will still mostly work. However Carmichael numbers are vanishingly rare, and the chances of hitting a $k$-bit one by accident are much less than $2^{-k/2}$ for sizable $k$ (perhaps more like $2^{-9k/14+o(1)}$, see this for more); thus for an RSA key with 512-bit factors, that's negligible.