How far is the list of known primes known to be complete?

Question

So there is always the search for the next "biggest known prime number". The last result that came out of GIMPS was $2^{74\,207\,281} - 1$, with over twenty million digits. Wikipedia also lists the twenty highest known prime numbers, only the four smallest on that list have fewer than three million digits.

For some while now, I have been wondering about the smaller prime numbers we haven't found. How far up is the list of known primes known to be complete? Since $500$ to $1000$ digit primes are considered safe for the RSA algorithm, I'd assume that it's well below that. How far along the number line have we checked that there are no more primes to be found? How fast is this boundary moving forward, currently? Have we, for instance, checked the primality of all numbers below $10^{100}$, or are we stuck somewhere south of $10^{20}$?

@YuriyS I don't want the list. I want to know how long the (longest) list (we can currently make) is. — Arthur, Jul 07 '16 at 10:51
Also of interest, what is the largest known prime which is not of the form $2^p-1$ (i.e., a non-Mersenne prime)? — barak manos, Jul 07 '16 at 10:53
@barakmanos According to wiki (the list I linked in the question) it is $19249×2^{13,018,586} + 1$, which is just shy of $4$ million digits, and there are $11$ known Mersenne primes that are larger. — Arthur, Jul 07 '16 at 10:55
Also of interest, what is the largest known prime which is not of the form $k\cdot2^n\pm1$ (where $n>1$ of course)? I bet that the list you're asking about doesn't go above this number. — barak manos, Jul 07 '16 at 10:59
A quote from above link > If we could give the smallest number n such that it is not known whether or not n is prime, then someone could check the next million primes in about a second of computer time (at most!). — Yuriy S, Jul 07 '16 at 11:00
@barakmanos Look no further than the wiki list: $475,856^{524,288} + 1$ . — Arthur, Jul 07 '16 at 11:00
@Arthur: Hmmmm... I guess that this pretty much rules out my bet then :) — barak manos, Jul 07 '16 at 11:01
@YuriyS: I have noted it. What I meant is, that your "about a second of computer time" estimation depends on how large that smallest number is. — barak manos, Jul 07 '16 at 11:02
@barakmanos (and others): The "also of interest" questions have filled the entire comment thread; may I suggest to delete these comments and start a new thread for those questions so that this one can stay focussed on the question posed? — joriki, Jul 07 '16 at 11:03
@barakmanos Any prime except 2 (and indeed any odd number) is of the form $k\cdot2^n\pm1$ with $n>1$. — Ivan Neretin, Jul 07 '16 at 11:03
@YuriyS I am aware that the boundary that I'm asking about moves much more frequently than the "largest known prime number" boundary, and also that I could, myself, bump that boundary up a few million in no time at all. However, the magnitude of the boundary doesn't change that fast, and seems to be between $10^{18}$ and $10^{19}$ according to your link. — Arthur, Jul 07 '16 at 11:04
@joriki: I agree, but I can only delete mine, which would leave this thread kind of enigmatic, so if you have the privilege of doing it, then please go ahead, I will not bare a grudge for that :) — barak manos, Jul 07 '16 at 11:05
Wiki also lists Wikipedia is a Wiki, but Wiki is not Wikipedia: https://en.wikipedia.org/wiki/Wiki — A.L, Jul 07 '16 at 12:36
Possible duplicate of Currently, what is the largest publicly known prime number such that all prime numbers less than it are known? — hmakholm left over Monica, Dec 23 '16 at 22:41
It has already be mentioned , but to again emphasize it : It is impossible to list all primes upto , say , $10^{50}$ , so the next unknown prime could be found out in less then a second (how long it actually takes is irrelevant). The question about the smallest not known prime is therefore pointless , and asking for the desired list is equivalent to this question. Small primes are just too easy to be found. — Peter, Jan 18 '23 at 10:51
@Peter And I'll invite you to read the second half of the final paragraph of my question to see more specifically what exactly I'm asking. I'm not really asking about an actual list stored on hard drives in some computer center. I'd say my question makes perfect sense, and although the exact answer could change by the millisecond, I'm mostly after a size estimate in the form of a natural power of ten or something similar, and that presumably doesn't really change that fast. — Arthur, Jan 18 '23 at 11:11

score 45 · Answer 1 · edited Jul 07 '16 at 15:49

The maximum of such list is far smaller than mentioned 500-digits. Due to the prime number theorem $\pi(x) \approx x/\log(x)$ so one could estimate that the list of prime numbers up to $x$ would require at the order of $x$ digits to represent.

So by using the sieve of Atkin the complexity is $O(x)$ for both time consumption and memory consumption. And all memory available is small enough to be feasible to traverse. This means that the memory available for storing such a list is what should be the limiting factor.

Basically this boils down to that the largest such list is as large as anybody have place (and need) for.

Now the total global storage is estimated to be at the order of $10^{21}$ bytes which means that the upper bound of such a prime number list is there. So there exist no complete list with primenumbers up to more than about twenty digits (and not even that since not all storage is devoted to store prime numbers).

Excellent line of reasoning. If you can't store it, you can't (continue to) know it. Astonishing remindee of the power of exponents... — Floris, Jul 08 '16 at 01:28
Suppose someone with extremely many huge hard disks kept a list of all prime numbers with at most 21 decimal digits. It would be extremely easy to come up with a continuation of that list. For example the first prime over $10^{21}$ (i.e. 22 digits) is $1{,}000{,}000{,}000{,}000{,}000{,}000{,}117$. It takes a few milliseconds to find and prove with standard math software (I used PARI/GP). — Jeppe Stig Nielsen, Jul 08 '16 at 08:26

score 24 · Answer 2 · answered Jul 07 '16 at 11:04

24

This may be a somewhat unsatisfying answer, but no-one's really keeping a complete list of known primes (to the best of my knowledge). Moreover, it's fairly easy to come up with large primes, and it's fairly easy to "guarantee" (guarantee being a slippery term), that a given large number is prime.

The Miller-Rabin primality test is an algorithm that takes a number $n$, and a "certainty" parameter $m$, and (in layman's terms) if $n$ is prime, it will return "PRIME". If $n$ is composite, it will almost certainly (again, a slippery term) return "COMPOSITE", but there's a small probability $(\frac{1}{4^m})$ that it will return "PRIME".

However, by setting $m$ high enough, to, say, something greater than 40, then it essentially means the probability of a composite number being declared prime is smaller than you winning the jackpot in the lottery twice in one week. Thus, for almost ALL practical purposes, it suffices to work with primes that pass the Miller-Rabin test to a high degree of certainty. Henri Cohen famously called such numbers "Industrial Grade Primes".

If you're still interested in having "proof" that a number is prime, may I suggest reading up on prime certificates. I haven't ever personally come across a situation in which you'd prefer a certified prime to an industrial grade prime however.

Finally, as a quick example, Mathematica can generate very large primes easily. The Mathematica command "RandomPrime[{10^1000, 10^1001}]" generates a random 1000 digit prime in 0.40625 seconds on my five year old desktop machine. This should give you some indication as to why mathematicians generally don't keep long lists of all known primes.

answered Jul 07 '16 at 11:04

MadMonty

918

4

so RSA uses "Industrial Grade Primes" right ? – Gabriel Romon Jul 07 '16 at 11:31
@LeGrandDODOM Yes, unless the given RSA implementation was written by some sort of masochist. – MadMonty Jul 07 '16 at 11:39
Is there even an algorithm that would work to actually prove that useful candidates are primes? I mean that the time complexity of such an algorithm might be prohibiting if the numbers were big enough to be useful for public key crypos. OK, kind of the time for proving that $p$ is prime is $O(\sqrt p)$, but the time for factoring $pq$ is $O(p)$ and one could have that $\sqrt p$ is within reach but $p$ isn't... – skyking Jul 07 '16 at 12:58
2

@skyking Best two I know of are: 1) the elliptic curve primality test, which is $O((\log(n))^6)$, on the condition that certain (probably true) conjectures are true, and 2) Miller's test, upon which the Miller-Rabin test is based, which is a deterministic test. It only works under the assumption of GRH, but it works in $O((\log(n))^4 \log^k((\log(n))^4)$, for some integer $k$. – MadMonty Jul 07 '16 at 14:23
5

"I haven't ever personally come across a situation in which you'd prefer a certified prime to an industrial grade prime"; one case might be if you're generating a prime for a long term DH group (e.g. IKE group 14) or EC curve (e.g. P256 or Curve25519); since such a group/curve might be used literally billions of times, it might make sense to go to the extra effort of doing the full certification. – poncho Jul 07 '16 at 16:54
@poncho Yes, I can believe that. – MadMonty Jul 07 '16 at 17:10
10

Are there any Industrial Grade Primes that are known to be composite? – Plutor Jul 07 '16 at 18:10
@Plutor So the Miller-Rabin test is probabilistic, so if you ran it twice on a composite number, there's a small chance the first time it would be flagged as prime, and the second time as composite. If you used a low value of $m$ however, it's not unfeasible that a composite $n$ would be marked as prime. – MadMonty Jul 07 '16 at 18:33
@MadMonty: Ah, now I see that the algorithm involves repeatedly picking a random 2<=k<=n-2. – Plutor Jul 07 '16 at 18:58
@Plutor Exactly. – MadMonty Jul 07 '16 at 19:07
2

@MadMonty The AKS primality test is deterministic, its correctness is proven unconditionally, and a variant of it has running time roughly $O((\log n)^6)$. – Sasho Nikolov Jul 07 '16 at 21:21
@SashoNikolov Yes, but the implied constant is so large that it's completely unpractical. – MadMonty Jul 07 '16 at 23:07
I agree AKS is not practical (and this has at least as much to do with the exponent 6 as with the constant). But I believe it answers @skyking's question whether there exists an algorithms that "actually" proves primality, and is much more efficient than the best known factoring algorithms. – Sasho Nikolov Jul 07 '16 at 23:34
You should always consider, that even using a perfect algorithm to certify a prime may produce a wrong result, if some bit-flips occur in the memory of the computer calculating it. This could be caused by hardware problems, or maybe even quantum fluctuations... So in the end in our physical reality even a perfectly certified prime can only be guaranteed to be 99.999....% safe. So with a big enough m a probabilistic test offers the same confidence as a perfect test. – Falco Jul 08 '16 at 12:56
@MadMonty I partly disagree with this. I have personally implemented Miller-Rabin, Solovay-Strassen and also the AKS primality test in various versions. While AKS is certainly much slower you are still able to use it to prove primality of quite big primes in a reasonable time (tens/hundreds of digits). The difference is that miller gives an almost certain answer in 2 seconds, while AKS takes like 3 minutes (which I consider reasonable), but not days or weeks. – Bakuriu Jul 08 '16 at 14:50
1

ECPP and APR-CL are the typical algorithms used for proofs of relatively large general form numbers. @Bakuriu, have you published your implementations or timings? "hundreds of digits in ~3 minutes" is many thousands of times faster than published results from other people. Your blog has a pre-v6 AKS implementation in Python pseudocode, which would take years for a 100 digit input. The fastest times I've seen use many algorithmic improvements and still take over 2 hours at 100 digits. APR-CL and ECPP complete a proof for this size in under 50 milliseconds. BPSW under 1 ms. – DanaJ Jul 09 '16 at 05:39

score 22 · Answer 3 · edited Apr 13 '17 at 12:20

22

It seems that this link, provided by Yuriy S in the comments above more or less addresses my question. It states that prime gap searches have checked the primality (but not stored the primes) for all numbers up to about 20 digits (the exact bound is always changing, since the numbers are relatively small).

edited Apr 13 '17 at 12:20

Community

1

answered Jul 07 '16 at 11:21

Arthur

199,419

8

Should've probably posted as an answer, since you got upvoted. Not that I mind though, glad I helped – Yuriy S Jul 07 '16 at 12:39
1

I'm not familiar with the algorithms used by the maximal prime gap project; however I'd think that if you're searching for a prime gap $> k$, you wouldn't need to check the primality of each integer. Instead, if you're at the verified prime $p$, the next integer you check is not $p+2$, but instead $p+k$ (and if that's not prime, work your way backwards). If you verify that (say) $p+k-6$ is prime, you don't need to check the primality of the other integers in the range... – poncho Jul 07 '16 at 17:01
I wish someone know, just an order of magnitude, how many primes there are with less than 20 digits. this QA is remarkably unmathematical :) Everyone's just saying "there's a lot!" and "there's a hell of a lot!" Geesh! – Fattie Jul 07 '16 at 20:18
1

@JoeBlow: according to the prime number theorem, there are approximately $10^{20} / \log 10^{20} \approx 2,171,472,409,516,259,138$ primes in that range. While this is just an approximation (and the lower dozen digits are almost certainly wrong), this is certainly within an order of magnitude. – poncho Jul 07 '16 at 20:23
Hmm. I think that's about 100 million terrabyte drives. (Is that right or did I stuff something?) These are tricky quantities to think about :) – Fattie Jul 07 '16 at 20:39
@JoeBlow - if you have $2^{18}$ numbers, and almost all of them have 20 digits, you need to store $4^19$ bytes, or 40 million TB. About 200 million laptops are sold every year source; if each laptop owner bought a 1 TB drive for backup, and you divided the available storage (a la SETI project) you'd comfortably store all those bytes in the unused disk space.... If you store two digits per byte (easy) you get twice the space. – Floris Jul 08 '16 at 01:36
2

@poncho People also counted the exact number of primes below $10^{20}$, it is $2{,}220{,}819{,}602{,}560{,}918{,}840$, see this Wikipedia subsection. That number divided by your number is about $1.02$ according to that table. This fraction tends to $1$ by the prime number theorem. – Jeppe Stig Nielsen Jul 08 '16 at 08:59
@Floris, thanks for that great calculation! – Fattie Jul 08 '16 at 13:49
hi @JeppeStigNielsen - ah, that is fantastic information. Thanks. I wish you would put it in as an answer. it's a nice thing to remember, there are about 2 million-million-million primes below 10^20. – Fattie Jul 08 '16 at 13:51
@Floris given the actual number - 2,220,819,602,560,918,840 - how many TB hard drives would we need? BTW you mention 2^18. It looks like it is 2.22 x 10^18. – Fattie Jul 08 '16 at 13:51
@JoeBlow I meant 2E18 not 2^18... Just being sloppy sorry – Floris Jul 08 '16 at 13:53

score 8 · Answer 4 · answered Jul 07 '16 at 19:43

Listing primes in order is a fairly trivial problem — in fact, I believe we have programs that can compute lists of primes faster than they can be written to disk, let alone displayed in any human readable format.

The thing is, there are a lot of primes. The entire internet put together probably couldn't store the list of all 20 digit primes.

But that's okay, because we don't need lists of primes — whenever we need primes, we can just generate them.

gnasher729 · Answer 5 · 2023-01-04T13:06:24.573

First, what is a "list of primes"? It's some fixed storage in a format that would allow me to know all members of an arbitrary set S of integers, for the special case that S is a subrange of the set of primes. For example "11, 13, 17, 19" is a list of primes. Given this list (and if you trust me) you can know four consecutive primes.

You can store a list of primes in a quite compact way. For example, just store the first prime in the list, followed by the differences between each prime and the next prime, compressed with a good compression algorithm. In my example, "11,2,4,2" would represent 11, 11+2 = 13, 13+4 = 17, 17+2 = 19. I would estimate that you can store a large list of primes using about 1 byte per prime.

Now I just bought a 4TB hard drive for about £70. So at a cost of £70 I can store a list of about 4 trillion primes. If I wanted to store the largest list I'd spend £100,000 (my wife would kill me) for 1,400 such hard drives and store a list of almost 6 quadrillion bytes. And that would probably be the largest complete list of known primes because ...

... because nobody in their right mind would do that! If you wanted the first billion primes greater than one quadrillion, I wouldn't read my list from one of these hard drives. I would create a prime sieve in the memory of my computer and calculate it on the spot. Cheaper and faster than reading a list.

You will find lists of special primes. Say "complete list of known primes of the form $2^n-1$ for some integer n ≥ 1". But storing a list of all primes is just nonsense.

PS. These 6 quadrillion bytes at a cost of £100,000, at about 1 byte per prime due to compression, could store a list of primes up to $120 \cdot 10^{15}$. I'd be willing to send you a list of primes up to $80 \cdot 10^{12}$ on a large hard drive for say $500, which would pay for some of the cost creating the list.

PPS. A "list" of known primes would have to be recorded. A number x that has at some point be determined to be a prime is not a "known prime" if it is not recorded. (Given a list of known primes, I can determine whether x is prime or not by checking that it would be in the list if it was prime due to the list size, and then checking whether it is on the list. If nobody recorded that x is prime then I cannot do this). And nobody records large lists of known primes because it is pointless. You wouldn't use a list to determine if x is prime.

A good point. And it addresses the question in the title (albeit in an indirect way), but not the more specific questions in the actual post itself. Like, for instance, the question "How far along the number line have we checked that there are no more primes to be found?" One could also argue that the fact that I said the list of known primes, I meant the more abstract list of all the natural numbers that we humans, through pen-and-paper calculation or computer algorithms or anything in-between have actually confirmed to be prime. — Arthur, Jan 04 '23 at 12:28
Certainly, this list changes all the time, but I would argue that the (also changing) "lowest prime not on that list" is still a quantity that can be reasoned about, at least in approximate terms. Such as how large it is (is it closer to $10^{20}$ or $10^{100}$?), and how quickly it grows (does it increase by an order of magnitude per year, slower, or faster?) — Arthur, Jan 04 '23 at 12:31

How far is the list of known primes known to be complete?

5 Answers5

Linked