Shouldn't complexity theory consider the time taken for different operations?

Question

I have read the answer found here which considers the size of integers when doing comparisons and how that affects on the basic cost of comparison.

I am trying to understand why each basic operation takes $O(1)$ time. For example branches in general are considered to run in $O(1)$ time when they usually don't.

In computer architecture branches are considered the most time-consuming and problematic part of the code and this is one of the reasons why a loop is unrolled by the compiler in order to reduce the amount of branches the processor needs to execute.

For example a simple branch (let's say if a>b) needs to do at least four things

1. Get a from memory (whether it is cache or RAM or on disk does not matter; cache misses and hits are a different field of study).
2. Get b from memory
3. Evaluate the two and select the proper answer
4. Search in instruction memory a new value for the Program Counter depending on the result of the evaluation.

Now get the variables from RAM is something done everywhere. So let's not count that. But since all modern processors load more than just one instruction from instruction memory (let's say for our case 100) so as to reduce instruction memory accesses and a branch usually leads to a different place in the instructions list (let's say if a>b call a method that swaps them), that would mean that branches take a lot more time than any other basic function.

Also in all modern processors multiplications and divisions take significantly much more time than additions and subtractions. Which is the reason why when multiplying or diving by numbers that are powers of two we should use logical bit shifting.

I know that in CS Theory and evaluating an algorithm we do not care. But in practice when implementing an algorithm the operations we use really effect the efficiency and complexity of our code.

So let's say we have three algorithms

1. $n$ additions
2. $n$ multiplications
3. $n$ branches

In Theory all three of these compute in time $n$ but in practice, 1 is faster than 2, which is faster than 3.

Answer 1

All complexity theory is relative to some model of computation. Typically, we consider models of computation where all basic operations take unit time because that's usually precise enough and, frankly, it's already difficult enough to do that kind of analysis.

Typically, running time analyses are only quoted asymptotically anyway. As long as there's a constant bound on the number of clock cycles required for any operation, saying that an algorithm takes $O(f(n))$ operations is exactly the same thing as saying that it takes $O(f(n))$ clock cycles: the difference is just in the constant hidden in the $O()$ notation.

Having said that, models of computation do vary in what operations they consider as being unit-cost. For example, ordinary Turing machines count moving the head one step as unit-cost; random access Turing machines can move the head farther at unit cost. People who study the complexity of matrix multiplication are often interested in the number of times you need to multiply matrix elements, rather than the total number of arithmetic operations, precisely because of your observation that multiplies are significantly more expensive than additions on a real CPU.

But, again, many things are swept under the carpet by asymptotic analysis. Suppose you did produce a model in which branches are much more expensive than other operations. Fine but, if you have a loop that needs to be iterated $n$ times, where $n$ is an input, your compiler can only unroll a constant number of iterations of the loop. So, instead of doing $n$ branches, you do, say, $n/8$ branches: that's still $O(n)$.

Answer 2

actually complexity theory has many "timing" and "cost" models but these tend to be more advanced and are not typically outlined at earlier/ introductory levels (where just the difficulty of asymptotic complexity analysis can be nontrivial or even difficult at times). some take into account the complexity of "low level" operations. for example

• arbitrary precision arithmetic. see eg analysis of the complexity of the RSA algorithm or factoring large numbers. here arithmetic operations scale roughly at $O(n)$ or $O(n \log(n))$ for the size of the numbers instead of "unit cost".

• there is a model called the RAM model of computing where their complexity is not exactly equivalent to other standard models eg Turing machines for array access etc.

• cache oblivious algorithms are designed to scale well with arbitrary size caches.

• even circuits can be considered a basic computation model that are not exactly identical in complexity to Turing machines & other models

there are many others! so the answer is yes, CS is a vast field now and has many theoretical models for the complexity of computation which are used according to context/ relevance.

Answer 3

What if someone came up with a new architecture (maybe something that is not based on transistor tech at all) which has different performance characteristics of the operations? For example: Multiplication is now as fast as addition and so on. Then you would need to create a new complexity theory for that particular architecture and update all the theories based on this new architecture - doesn't sound like theory anymore :)