3

If I understand correctly, the complexity of solving a computational problem is defined in terms of which instance $I$ of the problem, what size $n$ of the problem instance, and what algorithm $A$ for solving the problem instance.

Is the complexity of the problem at a given size $n$ of the problem instance defined as $$ \min_A \max_{I\in \{\text{instances of size }n\}} \text{complexity}(A,I,n)? $$ Note that the solution to the above optimization $A^*(n)$ and $I^*(n)$ are both functions of instance size $n$.

Or is the complexity of the problem defined for some same algorithm for all the problem instances and all sizes of the problem instances?

Or is the complexity of the problem defined for some same instance for all the algorithms that solve the problem and all the problem instance sizes?

Raphael
  • 72,336
  • 29
  • 179
  • 389
Tim
  • 4,875
  • 5
  • 36
  • 71
  • 1
    This question can probably be answered by picking up any textbook on the topic and reading the definition. – Raphael Oct 05 '12 at 13:43
  • 2
    @Raphael, is there a point in the above comment, other than dis-encouraging people to ask questions?! – Ran G. Oct 05 '12 at 14:41
  • 1
    It is intended to encourage better questions, which are more likely to be unearthed after a study of basic material. Questions like "how is defined" are "boring" (and that does not mean they should not be asked!); questions like "why is it defined thus?" or "Is definition equivalent?" are more interesting. – Raphael Oct 05 '12 at 19:34

1 Answers1

6

You are missing one thing, namely that the optimal algorithm and worst instance are not simply functions of the instance size. Also, I am not sure that minimising/maximising algorithm and instance independently of each other is doing the right thing.

The complexity of a problem is the complexity of an optimal algorithm solving the problem in the chosen machine model. It's as simple as that, but of course we have to define what "optimal algorithm solving the problem" means. Note that we have to fix the machine model (in particular what to count, see below), as complexities can differ between models.

First, "solving the problem" is clear: for all inputs (that are instances of the problem), the algorithm should give the correct answer, i.e. terminate and produce the correct result, or don't terminate. Let's denote the set of all algorithms that solve problem $P$ with $\newcommand{\Alg}[1]{\mathsf{Alg}_{#1}}\Alg{P}$.

Second, what is the "complexity" of an algorithm? There are many ways to investigate runtime; complexity theory has defaulted to worst-case time and Landau classes. Formally, we define the (worst-case) runtime function of some algorithm $A$ to be

$\qquad \displaystyle T_A(n) = \max \{ T_A(x) \mid x \in I_P, |x|= n \}$

where $I_P$ is the set of instances of the problem $P$ (which $A$ solves) and $T_A(x)$¹ is the runtime of $A$ on input $x$² (in clock cycles, state transitions, whatever fits your machine model). Now, the complexity of $A$ is the asymptotic growth of $T_A$. For example, if $T_A(n) = 17n^2 - 5n + 13$, we would say "The complexity of $A$ is $\Theta(n^2)$".

Finally, an optimal algorithm (for $P$) is an algorithm with minimal complexity (among all algorithms in $\Alg{P}$)³.

  1. We abuse notation here: depending on its parameter, $T_A$ means different things. It is usually done this way, though, and we don't need the runtime on individual inputs later on.
  2. Let's assume for the moment that we deal only with deterministic algorithms. Otherwise, there may be many different runtimes for a given input. In that case, we would choose the runtime of the longest computation; see here for reasons why.
  3. Note that Landau-$O$ implies a partial order on functions: $f \leq g \iff \exists c \in \mathbb{N}.\,\limsup\limits_{n \to \infty} \frac{f(n)}{g(n)} \leq c$.
Community
  • 1
Raphael
  • 72,336
  • 29
  • 179
  • 389
  • I would say that an an optimal algorithm for a problem $P$ is an algorithm whose complexity (measured using $O$ notation) matches the lower bound associated to the intrinsic complexity of the problem $P$ (measured using the $\Omega$ notation) if this lower bound is known, and, otherwise, an algorithm with minimal complexity among all of the known published algorithms solving $P$. In the latter case, the optimality is only temporary and a better algorithm may be published later. – Massimo Cafaro Oct 05 '12 at 15:12
  • Note that the definition I formulate does not contain any statement about "known" algorithms. If we don't know an optimal algorithm or we don't know whether the best known algorithm is optimal, we simply don't know the problem's complexity. That does not change the complexity, and it does not affect its well-definedness. (The "intrinsic" complexity equals the complexity of an optimal algorithm. In fact, it is defined that way; or how would you define it otherwise?) – Raphael Oct 05 '12 at 19:37
  • @Raphael With respect to footnote (2), what algorithm has infinite expected run-time? – Joe Oct 05 '12 at 21:49
  • @Joe I don't understand the question. The footnote does in no way relate to expected runtime. – Raphael Oct 05 '12 at 22:00
  • @Raphael It relates to expected running time because you suggested minimum running time as an alternative measure of running time. Consider flipping a coin until you get heads. What's a reasonable measure of the number of flips? 1 (the minimum) and $\infty$ (the theoretical maximum) both sound sub-optimal to me. – Joe Oct 05 '12 at 22:13
  • @Raphael: the intrinsic complexity of a Problem is not exactly defined that way indeed. It is an optimal algorithm that is defined that way: as an algorithm matching the lower bound for the problem. You forget that, to define the intrinsic complexity and therefore a lower bound, you need to specify a model of computation. For instance, sorting $n$ numbers using comparisons requires $\Omega(n lg n)$. If you change the model of computation, under suitable assumptions, you can sort in $\Omega(n)$. An optimal algorithm for a specific model of computation may not be optimal for another model. – Massimo Cafaro Oct 06 '12 at 06:20
  • @MassimoCafaro As you say, the "intrinsic complexity" is not well-defined per se, so I see no use using it in a definition. "The lower bound" is slippery again; do you mean the (maybe unknown) largest lower bound? If so, largest smaller bound on what? It seems as if you definition is circular. Your comment about machine models is fair, though; I'll clarify that aspect in the answer. – Raphael Oct 06 '12 at 09:39
  • @Joe I don't propose an alternative, but state how runtime is usually defined for nondeterministic machines. Using the average instead does not fit our intuition for nondeterminism -- the machine will not only guess a correct path, but also a shortest one. – Raphael Oct 06 '12 at 09:47
  • @Joe Also, averaging in case of nondeterminism is not meaningful; it is not the same as randomising. To humor you, let's assume it was and consider sorting nondeterministically by guessing the correct permutation, and looping if the guess is wrong. In that case -- assuming a uniformly random guess -- the probability of termination is $\frac{1}{n!}$ which results in "expected runtime" $\infty$. (There are less pathological examples of distributions with no finite expectation, e.g. Cauchy-distribution.) – Raphael Oct 06 '12 at 09:50
  • Thanks, Raphael! (1) For a problem, can it happen that the optimal algorithm is different for different values of $n$? In the definition of complexity of a problem, is the optimal algorithm chosen to be optimal for all values of $n$ according to some criterion, or different algorithm for different value of $n$? (2) Can an algorithm have more complexity than another algorithm for an instance $x_1 \in I_P$, and have less complexity for another instance $x_2 \in I_P$? If – Tim Oct 06 '12 at 22:12
  • @Tim Please re-read the definition of complexity I give and click through to the Wikipedia article about asymptotic growth. We compare algorithms only in the limit and up to a constant factor. Therefore, individual sizes let alone instances for which one algorithm is faster than another are irrelevant. (Of course, this changes if you investigate runtime optimality instead of complexity optimality.) – Raphael Oct 06 '12 at 23:24
  • @Raphael I missed the part where you were talking about non-deterministic machines. I thought you were talking about a deterministic algorithm running on different inputs. – Joe Oct 07 '12 at 06:40
  • @Joe I see. Is the footnote unclear, should I rewrite it? – Raphael Oct 07 '12 at 15:45
  • @Raphael: (1) "Let's assume for the moment that we deal only with deterministic algorithms. Otherwise, there may be many different runtimes for a given input." By "otherwise", do you mean randomized algorithms? (2) "averaging in case of nondeterminism is not meaningful; it is not the same as randomising", do you mean that deterministic algorithms running on a nondeterministic model of computation is different from a randomized algorithm? – Tim Oct 08 '12 at 04:44
  • @Tim 1) No, I mean nondeterminism. 2) Yes. – Raphael Oct 08 '12 at 08:17
  • @Joe I seem to have messed up the footnote after all; apparently the longest branch is used. – Raphael Oct 08 '12 at 08:20
  • @Raphael: Why are randomized algorithms and nondeterministic (machines or algorithms?) different? – Tim Oct 08 '12 at 14:10
  • @Tim By definition. Check those, and if you don't get it, that would make a good question for this site! – Raphael Oct 08 '12 at 20:05