86

Normally in algorithms we do not care about comparison, addition, or subtraction of numbers -- we assume they run in time $O(1)$. For example, we assume this when we say that comparison-based sorting is $O(n\log n)$, but when numbers are too big to fit into registers, we normally represent them as arrays so basic operations require extra calculations per element.

Is there a proof showing that comparison of two numbers (or other primitive arithmetic functions) can be done in $O(1)$? If not why are we saying that comparison based sorting is $O(n\log n)$?

I encountered this problem when I answered a SO question and I realized that my algorithm is not $O(n)$ because sooner or later I should deal with big-int, also it wasn't pseudo polynomial time algorithm, it was $P$.

Raphael
  • 72,336
  • 29
  • 179
  • 389
  • 5
    If you are going to count the complexity of comparing numbers, you should also write your complexity bounds in terms of the bit size of the input. So given $N$ $w$-bit numbers, the bit size of the input is $n=Nw$ and sorting can be done in $O(Nw \log N) = O(n \log n)$ time. – Sasho Nikolov Jun 09 '13 at 06:22
  • 3
    there are basically two "realms" or "regimes" of complexity study. generally $O(1)$ operations are assumed for "fixed-width" operations which is a reasonable approximation for most computer languages which have fixed-width number representations including for floating point eg 2-4 bytes (see eg IEEE standards). then there is "arbitrary precision arithmetic" where numbers have arbitrary size and there is more careful/precise study of the complexity of operations. the former context is more in applied analysis & the latter is more in theoretical/abstract analysis. – vzn Sep 14 '13 at 18:12

6 Answers6

85

For people like me who study algorithms for a living, the 21st-century standard model of computation is the integer RAM. The model is intended to reflect the behavior of real computers more accurately than the Turing machine model. Real-world computers process multiple-bit integers in constant time using parallel hardware; not arbitrary integers, but (because word sizes grow steadily over time) not fixed size integers, either.

The model depends on a single parameter $w$, called the word size. Each memory address holds a single $w$-bit integer, or word. In this model, the input size $n$ is the number of words in the input, and the running time of an algorithm is the number of operations on words. Standard arithmetic operations (addition, subtraction, multiplication, integer division, remainder, comparison) and boolean operations (bitwise and, or, xor, shift, rotate) on words require $O(1)$ time by definition.

Formally, the word size $w$ is NOT a constant for purposes of analyzing algorithms in this model. To make the model consistent with intuition, we require $w \ge \log_2 n$, since otherwise we cannot even store the integer $n$ in a single word. Nevertheless, for most non-numerical algorithms, the running time is actually independent of $w$, because those algorithms don't care about the underlying binary representation of their input. Mergesort and heapsort both run in $O(n\log n)$ time; median-of-3-quicksort runs in $O(n^2)$ time in the worst case. One notable exception is binary radix sort, which runs in $O(nw)$ time.

Setting $w = \Theta(\log n)$ gives us the traditional logarithmic-cost RAM model. But some integer RAM algorithms are designed for larger word sizes, like the linear-time integer sorting algorithm of Andersson et al., which requires $w = \Omega(\log^{2+\varepsilon} n)$.

For many algorithms that arise in practice, the word size $w$ is simply not an issue, and we can (and do) fall back on the far simpler uniform-cost RAM model. The only serious difficulty comes from nested multiplication, which can be used to build very large integers very quickly. If we could perform arithmetic on arbitrary integers in constant time, we could solve any problem in PSPACE in polynomial time.

Update: I should also mention that there are exceptions to the "standard model", like Fürer's integer multiplication algorithm, which uses multitape Turing machines (or equivalently, the "bit RAM"), and most geometric algorithms, which are analyzed in a theoretically clean but idealized "real RAM" model.

Yes, this is a can of worms.

Community
  • 1
JeffE
  • 8,703
  • 1
  • 36
  • 47
  • 4
    I know I'm just supposed to vote, but can't stop myself from saying it: This is the best answer. The trick is that (1) arithmetic operations are constant time by definition and that's OK because in theory you can choose any model, and (2) you should have some reasons for choosing a certain model, and this answer explains what they are. – rgrig May 04 '12 at 10:30
  • 1
    I agree with rgig, (Also I'm just supposed to vote), but a little problem is input size is not related to input numbers, e.g If I have an $n$ input my biggest number is $m$, and if I choose the computation model as the way I like, this causes to pseudo polynomial time algorithm became $P$, am I right? –  May 04 '12 at 11:03
  • 1
    If your input consists of numbers with more than $w$ bits, then to fit the model you have to split them into $w$-bit chunks, just like in real life. For example, if your input consists of $N$ integers between $0$ and $M$, then your true input size is $N\log_w M = (N\lg M)/(\lg w)$. Thus, pseudo-polynomial running times like $O(NM)$ time are still exponential in the input size when $M$ is big. – JeffE May 04 '12 at 11:48
  • Are there any algorithms analyzed in the Real RAM model that are not secretly "Order Type RAM" algorithms? I have never thought about it much, but can't quickly come up with an example that isn't. – Louis May 04 '12 at 19:45
  • 1
    @Louis: Yes, lots: Voronoi diagrams, Euclidean shortest paths, recursive cuttings, simplicial partition trees,.... But the best example is Gaussian elimination, which runs in $O(n^3)$ time on the real RAM model (or the unit-cost integer RAM, but needs $O(n^4)$ time on the integer RAM. – JeffE May 04 '12 at 21:32
  • @JeffE: Gaussian elimination is pretty much a canonical example of when one wouldn't get good predictions from the real RAM model, like I said in my answer elsewhere. For Voronoi diagrams, isn't the usual thing to compute the Delaunay triangulation with convex hulls one dimension higher? – Louis May 04 '12 at 21:55
  • @JeffE Why do you say that complexity theorists prefer the bit-Ram model? In most algorithm analysis one seems to completely ignore the issue with the size of each number involved -right? (like if an algorithm needs just one pass through the full data set you would always call it a "linear time" algorithm - right? - irrespective of how large is each number/data making up the data set - right? [...a curious exception seems to be the naive way of primality testing which needs $O(\sqrt{N})$ divisions but one wants to think of it as being exponential in the input size...] – user6818 Aug 27 '14 at 05:21
29

It just depends on the context. When dealing with bit level complexity of algorithms, we do not say that the addition of two $n$ bits numbers is $O(1)$, we say it is $O(n)$. Similarly for multiplication etc.

Massimo Cafaro
  • 4,247
  • 18
  • 27
  • From your referenced article: "can be measured in two different ways: one in terms of the integers being tested or multiplied, and one in terms of the number of binary digits (bits) in those integers", but this is not true, we should always measure by size of input. –  May 03 '12 at 09:04
  • 2
    @SaeedAmiri: it just depends on the encoding used. In the article, for instance, if the input is an integer $n$ specified using unary encoding, trial division will require only $\theta(n^{1/2})$. This is polynomial in the size of the input! Does this means that factoring by trial division is in $P$? No, the algorithm is pseudo-polynomial. Using the common binary encoding, you get an exponential algorithm, again in the size of the input. As stated, this happens because the number of bits in the input $n$ has become exponentially smaller changing its encoding. – Massimo Cafaro May 03 '12 at 11:45
  • 1
    By the way, pseudo-polynomial algorithms may actually be useful, if the order of magnitude of their parameters in actual instances is reasonably low. The most famous example is probably the pseudo-polynomial algorithm to solve the knapsack problem. – Massimo Cafaro May 03 '12 at 11:51
  • First I should mention that your referenced wiki page is not good because it doesn't have any good references, Also I don't know why you think I'm talking about pseudo-polynomial time algorithms, may be because input size normally is buttleneck in this cases? but I'm not talking about them, I'm talking mostly about problems which are in $P$ even by assumption of input size, like sorting, anyway because we can't cheating and say NPC problem is in $P$ I think we shouldn't say sorting is $O(n \log n)$ except we have a formal proof to ignore comparison. –  May 03 '12 at 12:01
  • I am discussing pseudo-polynomial algorithms to focus your attention on the size of the input, to show you that it can be misleading. Here is another example. You are given a natural number as input, say $n$, and the algorithm runs a loop in which it does $O(1)$ time operations for $n$ iterations. The complexity of this simple loop algorithm measured as a function of the input size, is $O(n) = O(2^{lg n})$. Since $lg n$ is the input size, the algorithm is exponential in the input size! Think about this. Now you may understand what I mean with "It just depends on the context". – Massimo Cafaro May 03 '12 at 12:41
  • Your last sample is exactly exponential time algorithm, I can't see anything wrong with this, If it wasn't exponential, people wont try to invent AKS, ... –  May 03 '12 at 13:00
  • Umhh... the majority of people would say that the previous example is a linear time algorithm, linear in $n$. But then, if you understand correctly what I mean, why you wonder that the majority of people claim that sorting is $O(n lg n)$ ? – Massimo Cafaro May 03 '12 at 13:41
  • @unfirgiven, you assumed I'm dumb? :))) but in $O(n\log n)$ case I didn't see any paper which says sorting is $O(n\log n \cdot \log m)$ my problem is this, I'm not talking about students, I'm thinking about well studied papers. –  May 03 '12 at 13:54
  • I have got your point now ;-) All of the papers I have read about discuss the worst-case complexity with regard to the number of comparisons done with respect to the input size $n$. The most complex paper on sorting, assuming additional information, I have read is: Y. Han. Deterministic sorting in $O(n log log n)$ time and linear space. Proceedings of the thirty-fourth annual ACM symposium on Theory of computing, Montreal, Quebec, Canada, 2002,p.602-608. Caveat: it's a very, very complex paper. – Massimo Cafaro May 03 '12 at 14:20
  • Yes I saw similar paper, but such papers assumes input size is bounded by computation model. I agree that normally in comparison base algorithms, they talk about number of comparisons (I should do some research around it). –  May 03 '12 at 20:20
18

To answer the question as stated: algorithmists just do it, fairly often, by using the RAM model. For sorting, in many cases, people even analyze the simpler comparison model, which I discuss a little more in the linked answer.

To answer the implicit question about why they do it: I would say that this model has pretty good predictive power for certain types of combinatorial algorithms, in which the numbers are all "small" and, on real machines, fit in registers.

To answer implicit follow-up about numerical algorithms: No, the plain old RAM model is not the standard here. Even Gaussian elimination can require some care. Typically, for rank computations, the Schwartz Lemma will enter (e.g., Section 5 here). Another canonical example is the analysis of the Ellipsoid Algorithm, which requires some care to analyze.

And finally: People have thought about string sorting before, even recently.

Update: The problem with this question is that "we" and "assume" are not so precisely specified. I would say that the people who work in the RAM model are not doing numerical algorithms or complexity theory (where determining the complexity of division was a celebrated result).

Community
  • 1
Louis
  • 2,926
  • 16
  • 25
9

I couldn't find any studies of this, but Kozen says in the introduction to "The Design and Analysis of Algorithms" that the $O(1)$ model "reflects experimental observation more accurately [than the log-cost model] for data of moderate size (since multiplication really does take one unit of time)." He also gives a reference to this paper as an example of how the $O(1)$ model can be abused.

This is absolutely not a legit evaluation (not least because it's Python), but here's some numbers from running python -mtimeit "$a * $b" for $a in $10^{\{1,2,...,66\}}$ and $b = 2*$a. (I stopped at 66 because that's when Python syntax stops accepting integer literals and I'd have to slightly switch my evaluation code, so I didn't. :p)

Each number is the mean of 10,000,000 loops, where it takes the best time of 3 runs in each loop. I'd do error bars or something but that'd be more effort. :p In any case, it looks pretty constant to me, even out to $10^{50}$ -- slightly surprising, since $\log_{10}(\tt{sys.maxint})$ is 43, which reinforces my suspicion that maybe this evaluation is particularly bogus and I should do it in C.

Juho
  • 22,554
  • 7
  • 62
  • 115
Danica
  • 191
  • 4
  • May be in work experience in this special case you are right (but I'm not sure:), but e.g take a look at this, seems it's $O(n)$ but it's not, it also is a practical problem. Also did you see any paper which saying sorting is $O(n \cdot \log n \cdot \log m)$? –  May 03 '12 at 09:15
9

You are right, in general we cannot assume they are $O(1)$.

Strictly speaking, if we want to sort an array with N numbers using comparisons, and the largest number is M, then on the worst case, each comparison may involve $O(\log M)$ comparisons on the bit level. And if our algorithm does $O (N \log N)$ comparisons, then its total complexity will be $O (N \log N \log M)$.

However, you will notice the difference only for very large values of $M$, that cannot be stored in a single register, as you can see from Dougal's experiment.

Erel Segal-Halevi
  • 5,994
  • 1
  • 23
  • 59
5

I would say that we typically assume O(1) arithmetic operations because we're usually doing things in the context of 32-bit integers or 64-bit integers or IEEE 754 floating point numbers. O(1) is probably a pretty good approximation for that kind of arithmetic.

But in general, that's not true. In general you need an algorithm to perform addition, subtraction, multiplication and division. Boolos, Burgess and Jefferies' Computability and Logic springs to mind as a way to understand the proof(s) of that, in terms of a couple of different formal systems, Recursive Functions and Abacus Machines, at least, in my 4th Edition copy.

You can look at the lambda-calculus terms for subtraction and division with Church Numerals for an easy-to-see explanation of why those two operations aren't O(1). It's a bit harder to see for addition and multiplication and exponentiation, but it's there if you consider the form of Church Numerals themselves.

Bruce Ediger
  • 566
  • 2
  • 7