Edit [2014-10-25]: Due to respect for K. Knopp (and also for my pleasure) I bought the english translation Theory of Functions and replaced in the answer below my translation of the German classic Funktionentheorie I (1949) with an authorized version.
A few more aspects (and a word of warning at the end)
- Statements from old masters
Many of the answers above provide valuable information around the question what's so special about these holomorphic functions.
I was curious to find statements from the old masters which could help to provide us with some more insight. I had a look for some classics about complex analysis like those of A.Hurwitz, W. Rudin, J.B. Conway, S. Lang and some more.
For me the most valuable statement was given by K.Knopp in his classic Theory of functions. The interesting thing is that in this book there are some places where he refers to our question. It culminates in the section 7.21 The Identity Theorem for Analytic Functions:
Konrad Knopp (extract from Theory of functions)
Now, it is exceedingly remarkable that by means of the single requirement of differentiability, that is, the requirement of regularity, a class of functions having the following properties is selected from the totality of the most general functions of a complex variable. On the one hand, this class is still very general and includes almost all functions arising in applications. On the other hand, a function belonging to this class possesses such a strong inner bond, that from its behavior in a region, however small, of the $z$-plane one can deduce its behavior in the entire remaining part of the plane.
...
A first theorem in this direction is Cauchy's formula which enables us to deduce the values of the function in the interior of a simple closed path $\mathcal{C}$ from the values along the boundary. A second result of this kind is the statement made in connection with the expansion theorem as to the magnitude of the true circle of convergence of a power series. Indeed, here we have already taken into consideration points of the plane which do not even belong to the original region of definition of the function.
... we are now in a position to derive a result ... which is because of its great importance for the development of the theory of functions, the most fundamental result besides Cauchy's integral theorem.
The identity theorem for analytic functions: If two functions are regular in a region $\mathcal{G}$, and if they coincide in a neighborhood, however small, of a point $z_0$ of $\mathcal{G}$, or only along a path segment, however small, of a point $z_0$ of $\mathcal{G}$, or also only for an infinite number of distinct points with the limit point $z_0$, then the two functions are equal everywhere in $\mathcal{G}$.
Observe the dramatic construction of this theorem which Knopp uses to even more point out the significance of it! :-)
Note: The formulation of the translated version is ... most fundamental result after Cauchy's integral theorem .... In my opinion this is not correct. The formulation in the german original is neben which means besides (implying an equal ranking, and not a ranking after resp. behind Cauchy's integral theorem).
So, for me the essence of the nice behavior of holomorphic functions is that they are already characterized within a region $\mathcal{G}$ by infinitely many different points converging at an accumulation point. The situation is similar to a polynomial of degree $n$ which is already completely determined by $n+1$ different points, irrespectively how near or far they are.
And the magic is that all that can be derived from solely requiring differentiability of a function within a region $\mathcal{G}$.
Observe that also the topology namely the region $\mathcal{G}$ is an essential ingredient in this statement.
And now for something completely different:
Similar questions in MSE
There are questions in MSE similar to this one (of course :-)) and I would like to emphasize two of them:
The answers in Why are differentiable complex functions infinitely differentiable? are interesting, since some of them also try to provide an intuitive feeling.
A fundamental difference between complex and real differentiability is the type of linearity. Approximation of complex functions by a $\mathcal{C}$-linear map in contrast to a approximations of real functions by $\mathcal{R}$-linear maps is the reason that the Cauchy-Riemann differential equations hold in the complex case and providing so the bases for the miracle of the nice behaviour of holomorphic functions. See e.g. How is $\mathcal{C}$ different than $\mathcal{R}^2$?
Word of warning
This is not a part of the answer but a remark and also a word of warning to a reference from the answer of @pbs to the Wolfram page Complex Differentiable.
First I have to admit that many, many pages of Wolfram are a highly valuable source of information and I really appreciate this service. But regrettably sometimes a page is not sufficiently elaborated. And the page Complex Differentiable is one of these rare events.
Three critical aspects:
For me it's annoying that this page creates the impression of presenting a definition of complex differentiable and also referring thereby to the author G. Shilov and his book Elementary Real and Complex Analysis so that the reader could think this definition is cited from this book. This is not the case!
- Definition of complex differentiable: Shilov defines this term in section 10.11.a: Differentiation in the complex domain as follows
Let $C_z$ be the plane of the complex variable $z=x+iy$,$\ldots$, and let $C_w$ be the plane of the complex variable $w=u+iv$,$\ldots$. Then a function $w=f(z)$ defined on a set $E\in C_z$ and taking values in $C_w$ is called a (complex) function of a complex variable.
Let $z_0\in E$ be a nonisolated point of $E$ so that every neighborhood of $z_0$ contains a point of $E$ other than $z_0$ itself. Then a complex number $A$ is said to be the derivative of the function $w=f(z)$ at the point $z=z_0$ relative to the set $E$, denoted by $f^\prime_E(z_0)$, if, given any $\varepsilon>0$, there exists a $\delta>0$, such that $0<|z-z_0|<\delta,z\in E$ implies
$$\left|A-\frac{f(z)-f(z_0))}{z-z_0}\right|<\varepsilon$$
In this case, we say that $w=f(z)$ is differentiable at $z=z_0$, with derivative $A=f^\prime_E(z_0)$, where $A$ is clearly the limit of the difference quotient $\frac{f(z)-f(z_0)}{z-z_0}$ in the direction $z\rightarrow_{E} z_0$ determined by the set of intersections of $E$ with all deleted neighborhoods $0<|z-z_0|<\delta$.
The satisfaction of the Cauchy-Riemann equations which Wolfram cites follows in Shilov's book later from the definition, thereby precisely distinguishing between the different domains of the function under consideration and the different consequences resulting thereby!
- complex differentiable vs. holomorph:
Despite the statment in Wolframs page, the term complex differentiable is per se NOT THE SAME as holomorphic (or analytic or regular, which may be used interchangeably) and G. Shilov states this clearly in his book. He defines in the section 10.11.f:
A function $f(z)$ is said to be analytic (synonymously, holomorphic or regular) on an open set $G \in \mathcal{C}_z$ if $f(z)$ is differentiable on $G$.
Please note, that we have to clearly distinguish between phrases $f$ is complex differentiable in $z_0$ and $f$ is complex differentiable in $\mathcal{G}$!
The third aspect adresses Wolfram's statement: $f:z\rightarrow\overline{z}$ is not complex differentiable. It would be helpful if Wolfram had pointed out the conditions which are necessary that this statement is true!
We can read in Shilov section 10.12.b:
- The function $f(z)=\overline{z}=x-iy$ is differentiable at every point $z=z_0$ relative to any ray $E$ drawn from $z_0$, since
$$\frac{f(z)-f(z_0)}{z-z_0}=\frac{\overline{z-z_0}}{z-z_0}=-2\mathop{arg}(z-z_0)$$
and hence
\begin{align*}
f^\prime_E(z_0)=-2\mathop{arg}(z-z_0)\qquad\qquad(z\in E,z\ne z_0)\tag{2}
\end{align*}
However,(2) shows that $f(z)=\overline{z}$ fails to be differentiable relative to any set containing two distinct rays drawn from $z_0$. Therefore $f(z)=\overline{z}$ fails to be analytic at every point $z=z_0\in C_z$.
Conclusion: We should always check at least two independent sources when learning new terms.