According to the Wikipedia article,
[Euler] introduced much of the modern mathematical terminology and notation, particularly for mathematical analysis, such as the notion of a mathematical function.
— Leonhard Euler, Wikipedia
What was the notation for functions before him?
I scanned through parts of Newton's Pricipia found online, and was surprised that a search for the word "function" did not yield any results at all. There do appear to be equations acting as what we would call functions, such as when he describes force, we see things such as $$F=\frac {2h^2}{SP^2}\cdot \frac {QR}{QT^2}$$ and $$R=\frac {\frac 1 2 L}{1+e\cos ASP}$$ (on page 223) but he refers to these as equations, not functions, and admittedly (written the way they are) that is exactly what they are. It seems anything that we would today write as a function, Newton described in words, such as:
If a hyperbolic orbit be described under the action of a repulsive force tending from the center, the force varies as the distance and the velocity at any point as the diameter of the conjugate hyperbola parallel to the tangent at the point.
Or he used words within his equation:
$$\text{Velocity at P}=\frac {h.VA}{SP^2}$$
This last one almost assuredly would be written as a function if presented in a modern textbook. Newton is certainly not the only source one should consider, but it does give an idea of what was going on right before Euler began publishing.
The information on this website, which unfortunately does not include specific sources, indicates that Bernoulli proposed that $\phi$ or $\phi x$ be used as the notation for a function, and Euler introduced $f(x)$.
Edit: Reference #11 from David Renfro's answer gives references for the statements made about Bernoulli and Euler on the website, as described in the last paragraph above. In my brief skim of Newton's Principia, I also found exactly what was described in reference #11 to be true, specifically that the arguments were motivated almost exclusively from analytic geometry, and that what we consider a "function" was really only considered a variable, as is indicated in the few examples above. I would recommend reading #11, it explains in good detail what you would like to know, I think.
When I get a chance (I've been extremely busy at work the past couple of weeks), I'll look through the following references that I have copies of and see what I can find. However, I thought I'd post the list of references in case others might be interested. This is not intended in any way to be complete, but rather it only represents the items that I happen to have copies of in a binder devoted to secondary literature on the history of functions. The references are listed in chronological order of their first publication date.
[1] Lloyd Lyne Dines (1885-1964), The development of the function concept, School Science and Mathematics 19 #2 (February 1919), 99-110. [not reviewed by JFM]
http://books.google.com/books?id=X11LAAAAMAAJ&pg=PA99
[2] George Abram Miller (1863-1951), The development of the function concept, School Science and Mathematics 28 #5 (May 1928), 506-516. [JFM 54.0006.06]
[3] Herbert Russell Hamley (1883-1949), The history of the function concept, Chapter 4 (pp. 48-84) in Relational and Functional Thinking in Mathematics, 9th Yearbook, The National Council of Teachers of Mathematics, 1934, viii + 215 pages. [JFM 60.0864.03; not reviewed by Zbl]
[4] Nikolai Luzin [Lusin] (1883-1950), Function: Part I, translation of the 1934 Russian version by Abe Schenitzer, American Mathematical Monthly 105 #1 (January 1998), 59-67. [MR 2000k:01055; Zbl 913.01015]
[5] Nikolai Luzin [Lusin] (1883-1950), Function: Part II, translation of the 1934 Russian version by Abe Schenitzer, American Mathematical Monthly 105 #3 (March 1998), 263-270. [MR 2001b:01031; Zbl 916.01029]
[6] Carl Benjamin Boyer (1906-1976), Historical stages in the definition of curves, (National) Mathematics Magazine 19 #6 (March 1945), 294-310. [MR 6,253o; Zbl 60.00304]
[7] Carl Benjamin Boyer (1906-1976), Proportion, equation, function: Three steps in the development of a concept, Scripta Mathematica 12 (1946), 5-13. [MR 8,126f; Zbl 63.00581]
[8] Salomon Bochner (1899-1982), The rise of functions, pp. 3-21 in H. L. Resnikoff and R. O. Wells (editors), Complex Analysis, 1969 (Proceedings of the Conference on Complex Analysis, Rice University, 26-29 March 1969), Rice University Studies 56 #2 (Spring 1970), vi + 222 pages. [MR 44 #6428; Zbl 248.01001]
[9] Antonie Frans Monna (1909-1995), The concept of function in the 19th and 20th centuries, in particular with regard to the discussions between Baire, Borel and Lebesgue, Archive for History of Exact Sciences 9 #1 (1972), 57-84. [not reviewed by MR; Zbl 249.01008]
[10] Adolphe Pavlovich Youschkevitch (1906-1993), The concept of function up to the middle of the 19th century, Archive for History of Exact Sciences 16 #1 (1976), 37-85. [MR 58 #15925; Zbl 362.26006]
[11] Israel Kleiner (??- ), Evolution of the function concept: A brief survey, College Mathematics Journal 20 #4 (September 1989), 282-300. [not reviewed by MR; not reviewed by Zbl]
http://mathdl.maa.org/images/upload_library/22/Polya/07468342.di020738.02p00875.pdf
[12] Jesper Lützen (1951- ), Between rigor and application. Developments in the concept of function in mathematical analysis, Chapter 24 (pp. 468-487) in Mary Jo Nye (editor), The Cambridge History of Science, Volume 5 (The Modern Physical and Mathematical Sciences), Cambridge University Press, 2003. [not reviewed by MR; Zbl 1059.01007]
The question is ill-posed, as there was no concept of function before Bernoulli and Euler in anything approaching the modern sense.
Leibniz and Bernoulli mostly worked with variable quantities. Thus, a curve may be defined by an equation involving both $x$ and $y$ and the functional relation between $x$ and $y$ was only implicit, because neither was chosen as the independent variable upon which the other would depend. Similarly, what we would today call differentiation involved relations among infinitesimal differentials $dx$ and $dy$. What we would today express as the idea of independent variable was expressed by saying that $dx$ varies in constant increments.
Since there was no concept there could not have been a notation for it. Thus, in the 17th century mathematicians mainly worked with curves defined by an equation, and studied there properties. This does not require anything close to the modern notion of a function and they did not have one, though Leibniz did realize the functional relationship between $x$ and $y$ along a curve.
Let's observe an example :
$a)$ formal description of function (two-part notation)
$f : \mathbf{N} \rightarrow \mathbf{R}$
$n \mapsto \sqrt{n}$
$b)$ Euler's notation :
$f(n)=\sqrt{n}$
I don't know who introduced two-part notation but I think that this notation must be older than Euler's notation since it gives more information about function and therefore two-part-notation is closer to correct definition of the function than Euler's notation.
There is also good wikipedia article about notation for differentiation.
The arrow, e.g., →, was developed for function notation in 1936 to denote images of specific elements[note 95] by Øystein Ore.[note 96] Later, in 1940, it would take it's the present form, e.g., f: X → Y, through the work of Witold Hurewicz. Werner Heisenberg, in 1941, proposes the S-matrix theory of particle interactions.
– Pod Oct 22 '13 at 13:31