27

Given any computational problem, is the task of finding lower bounds for such computation really possible? I suppose it boils down to how a single computational step is defined and what model we use for the proof, but given that, do we really prove a lower bound then in general? What I means is that can we prove something like "problem $X$ cannot be solved faster than $t(X)$ time" rather than "problem $X$ can be solved in $t(X)$ time or faster"?

I have tried to find information specifically on lower bounds and proofs of them, but I can't really find any of interest, any recommendations on books/papers/websites on the subject?

Raphael
  • 72,336
  • 29
  • 179
  • 389
hsalin
  • 733
  • 7
  • 7

3 Answers3

22

We can absolutely prove such things.

Many problems have trivial lower bounds, such as that finding the minimum of a set of $n$ numbers (that are not sorted/structured in any way) takes at least $\Omega(n)$ time. The proof for this is simple: a hypothetical algorithm that runs in $o(n)$ time can not examine all of the numbers in the input. So if we ran the algorithm on some input, we could observe that it never examined a particular element of the input. Changing that element to the minimum, we can get the algorithm to fail.

A less trivial lower bound is the $\Omega(n \log n)$ lower bound for sorting in the comparison-based model. The proof for that goes along the following lines: given an input of $n$ numbers, there are $n!$ possible outputs (the input could be any permutation of the sorted list, so the output can also be any permutation of the input). If we are limited to only doing comparisons, then our algorithm (on average) needs to perform at least $\log_2(n!)=\Omega(n \log n)$ comparisons in order to be able to give $n!$ different outputs.

Lower bounds can be even stronger. There are several problems (notably the $EXPTIME$-hard problems) for which there is an exponential lower bound. Problems in this class include computing optimal strategies for games such as (generalized) chess, checkers and go. The proof of this is via the Time Hierarchy Theorem, which states (subject to some restrictions on $f$):

Given a function $f$, there exists a computational problem that can be solved in time $O(f(n))$ but can not be solved in time $o(\frac{f(n)}{\log n})$.

So basically, if you can think of a function $f$ there exists a problem that requires that much time to solve.

Finally, another avenue of not necessarily proving a time lower bound but something even stronger is showing the undecidability of a problem (e.g. halting, post correspondence).

Tom van der Zanden
  • 13,238
  • 1
  • 35
  • 54
15

Yes, it's possible. The classic example is the fact that any comparison-based sorting algorithm requires $\Omega(n\log n)$ comparisons to sort a list of length $n$.

However, lower bounds seem to be much harder to prove than upper bounds. To prove that there's a sorting algorithm that requires $O(n\log n)$ comparisons, you just need to exhibit such an algorithm (merge sort – voila!). But for a lower bound, you somehow need to show that no algorithm in a particular class can solve your problem. The difficulty of doing that is illustrated by the fact that we only know that $$\mathrm{L}\subseteq \mathrm{NL}\subseteq \mathrm{P}\subseteq \mathrm{NP} \subseteq\mathrm{PSPACE}\,,$$ even though we know that at least one of those inclusions is strict ($\mathrm{L}\subset \mathrm{PSPACE}$ by the space hierarchy theorem) and most people think they are all strict.

On the other hand, Ryan Williams has a nice paper (and talk, which I've heard a couple of times) called Algorithms for Circuits and Circuits for Algorithms, in which he argues that finding lower bounds and finding algorithms aren't fundamentally all that different. For example, he cites the proof of the undecidability of the halting problem as an example of an algorithm (the universal Turing machine) being used exactly to prove a lower bound (undecidability).

David Richerby
  • 81,689
  • 26
  • 141
  • 235
  • 1
    I think this is what I'm after "..you somehow need to show that no algorithm in a particular class can solve your problem.", this is the part I find a bit confusing since I cannot really intuitively see how one could do such thing, in general at least. As @Tom van der Zanden described the finding minimum number I do understand, but is that approach general? I mean general as in having that sort of argument when constructing the proofs?

    Thanks for the link as well.

     – hsalin Feb 14 '15 at 13:12
  • 1
    @user1288420 You're not alone. If anyone could see intuitively how to prove that no algorithm in a particular class can solve some problem, we'd have many more lower-bound results! Typically, you need to come up with some property that every algorithm in the class has, and show that that property prevents some problem being solved. For example, every Turing machine that runs in sublinear time has the property that it can't even read all of its input. That means it can't solve most problems. That's trivial; unfortunately, more interesting cases seem to be impossibly hard. – David Richerby Feb 14 '15 at 14:40
3

To take a trivial example, you cannot find the largest number from a set of $n$ numbers without checking them all which requires linear time. No big proof. But there are algorithms that do not always require reading all the data. A nice example is searching all occurences of a pattern in a string, which may not require reading the whole string (Boyer-Moore algorithm). But I should not repeat what is already answered, probably better than I would.

However, there is a point in the question that calls for some more remarks regarding lower bound (or complexity bounds in general).

Actually the choice of what is a single computational step is irrelevant, as long as computational steps can be considered as having a constant upperbound (and lower bound). The complexity result will be the same since it is defined up to a constant. Taking 3 comparisons as unit operations, or only a single one, makes no difference.

The same is true regarding the size of data which serves as reference to evaluate the cost of the computation. Taking a single integer, or two integers as unit of size makes no difference.

However, the two choices must be related.

You may consider that a number $n$ is a single unit of data if you consider that operations on integers, such as addition or comparison, are unit operations. You may also consider that it is $\log n$ units of data since it takes $O(\log n)$ digits to represent a number. Then, of course, addition and comparison are no longer unit operations, and their cost depends on the values of the operands.

Whether an operation can be considered having unit cost is tightly related to what data can be considered as having unit size. And that depends on what level of abstraction you choose for your model of computation.

babou
  • 19,445
  • 40
  • 76
  • Finding all occurrences of a pattern in a string trivially requires reading the whole string: if the pattern is "a", you can't find all occurrences without checking whether every single character of the string. – David Richerby Feb 14 '15 at 12:34
  • 1
    @DavidRicherby Actually not always. The Boyer-Moore algorithm start from the end of the pattern, thus jumping up in the string. If the attempted match fails, it does not have to read the beginning of the string. And it also has optimization similar to the Knuth-Morris-Pratt algorithm to skip attempts that are dommed to fail because of the structure of the pattern. Of course, there are patterns that do require reading the whole string. – babou Feb 14 '15 at 13:29
  • @DavidRicherby Why did you ask, you knew that? – babou Feb 14 '15 at 15:22
  • I don't understand your second comment. Your original answer contained an incorrect claim. Of course I knew it was incorrect: that's how I was able to point it out! Other people might not have known it was incorrect, so it would have been confusing to them to leave the answer as it was. – David Richerby Feb 14 '15 at 15:49
  • 1
    @DavidRicherby My point is you understood what I meant. I should have said may not rather than does not. This did not call for a comment style making readers believe I was talking nonsense. And while doing so, you made exactly the same careless mistake: by stating "Finding all occurrences of a pattern in a string trivially requires reading the whole string", when you should have said "Finding all occurrences of a pattern in a string may* require reading the whole string*". I only intended to state that reading the input may not always be necessary, to mitigate my previous example. – babou Feb 14 '15 at 16:59
  • I'm sorry you objected to the way I phrased my comment. But, now that you've brought me back here, I realise that your answer is still incorrect. It is impossible to find all occurrences of any pattern in a string without scanning the whole string. Until you've read the whole string, you don't even know where the input ends so you cannot possibly say there are no more matches. Things like Boyer-Moore and KMP don't let you avoid reading the whole string,: they just let you avoid looking for matches of the pattern starting at every position in the string. – David Richerby Feb 14 '15 at 17:15
  • @DavidRicherby Your first comment was rude, uncalled for, and erroneous with precisely the same (linguistic) mistake you were reproaching me. In the last comment, assuming, as usual, that the string is given as an array, your statement about Boyer-Moore algorithm is simply wrong. You apparently do not understand it. – babou Feb 15 '15 at 17:45
  • You are talking about different notions of "always", I think. @DavidRicherby states (correctly) that there is no string matching algorithm that can for every text and every pattern work without reading the whole text. That's the worst-case point of view. babou states (correctly) that there are some combinations of pattern and text for which these algorithms don't have to read the whole input. I seem to remember that this is even true in the average-case for some algorithm. – Raphael Feb 16 '15 at 10:26
  • Both, please be nice. babou, making the answer as clear as possible should be in all of our interest. David, sometimes a straight-forward request/proposal for improvement is a good option. – Raphael Feb 16 '15 at 10:32