For each of the functions you mention, the answer to "why it's useful" will be somewhat different. In general, if a function appears in many unrelated contexts, that's a sure sign that it's useful. Think of other mathematical structures that appear in many unrelated contexts: the natural numbers, the real numbers, the complex numbers, polynomials, trigonometric functions, etc.
My understanding is that in the olden days, the second half a complex analysis course - after the theory has been developed - was a detailed study of special functions. (At least this may have been the case in courses for physicists.) That doesn't seem to be so common anymore.
Some remarks about specific functions that you mention:
Gamma function: Just as it turned out to be very convenient to allow the $x$ in $a^x$ to take non-natural-number values, it is convenient to allow the $n$ in $n!$ to do so. To give but one small example of an occurrence of the Gamma function in the answer to a naturally occurring question, the volume of an $n$-dimensional ball is
$$
V_n(r)=\frac{\pi^{n/2}r^n}{\Gamma\left(\frac{n}{2}+1\right)}.
$$
This example may not be altogether convincing, because the Gamma function can be eliminated in favor of more elementary functions, but it does have the advantage of providing a uniform formula for odd and even $n$, which the more elementary formulas to not. This is just the tip of the iceberg as far as appearances of the Gamma function in mathematics and physics go.
Functions defined in order to compute integrals: If the $\ln$ function were not already known, it would have been necessary to invent it in order to carry out $\int\frac{1}{x}\,dx$. The inverse of the the $\ln$ function would have then led to the exponential function. Similarly, if the inverse trigonometric functions, $\arctan$, $\arcsin$ were not already known, it would have been necessary to invent them in order to carry out integrals such as $\int\frac{1}{\sqrt{1-x^2}}\,dx$, $\int\frac{1}{1+x^2}\,dx$. This would then have led to the creation of $\tan$ and $\sin$.
In the case of elliptic functions, this actually is what occurred. The computation of the arc length of an ellipse (which is of interest in the study of planetary motion) leads to an integral that cannot be expressed in terms of elementary functions. This led to the definition of "elliptic integrals", whose inverses are elliptic functions, to which the theta functions are closely related.
Elliptic functions have the remarkable property of double periodicity. That is, they are periodic in two directions in the complex plane. You can imagine how this might be useful in the study of planar lattices, for example. But more generally, any time you have a function whose domain of definition is the torus, you will find that elliptic functions come into play. Certain complex algebraic curves of degree three are topologically tori; elliptic functions can be used to parametrize positions on such curves, much as $\sin$ and $\cos$ parametrize positions on the circle.
Solutions to differential equations: One of the first classes of differential equations you come to whose solutions cannot be expressed in terms of elementary functions are second order linear differential equations whose coefficients are polynomials of low degree. Hence the Airy functions, Bessel functions, hypergeometric functions, and so on. The simpler the differential equation, the more often it will come up. The differential equations that lead to the above functions arise repeatedly in classical mechanics, electromagnetism, and quantum mechanics.
Orthogonal polynomials: Principles of linear algebra can be brought to bear on the study of spaces of functions. Just as it is useful to write down an orthonormal basis for $\mathbf{R}^n$, it is useful to write down a basis for certain spaces of functions. Depending on the domain and symmetries of the functions of interest, different orthogonal polynomials will come into play. Just as periodic functions may be expanded as Fourier series in $\sin$ and $\cos$, series with other symmetries may be expanded in terms of other types of functions. Often the coefficients of the lowest order terms in the expansion have clear physical meaning. You mention the Chebyshev polynomials. Another example are Zernike polynomials, whose domain of definition is the disk, of which an example is the pupil of the eye. In optometry, the coefficients of some of the low order polynomials relate to certain refractive errors in the eye. In physics and chemistry, the spherical harmonics are used to describe atomic orbitals. They are useful in many other problems with spherical symmetry, including celestial mechanics.
Zeta function regularization is used as one possible means of regularization of divergent series in quantum field theory. In one notable example, the Riemann zeta-function shows up explicitly in the calculation of the Casimir effect. The zeta function is also useful for the analysis of dynamical systems, see [19].
– Pedro Apr 27 '12 at 05:51