I know that the notion of "set" is one that cannot be defined mathematically since it is the fundamental data type that is used to define everything else (and the definition which says that "sets" are the objects in any model of set theory is to me circular since models are defined in terms of sets).
It seems to me that there is another fundamental concept just like "set", namely the notion of a "variable". Is this true?
dtldarek has given an answer from one point of view. Let me offer another.
A variable in mathematics often means an element in something (a ring, a group, a vector space, ...) which can be specialized to some more specific value.
In modern algebra, this notion becomes formalized in various ways, one of which is by the notion of the free object on (wikipedia say "over", but my own experience is that it is more common to speak of the free object "on") a particular set of variables.
If you haven't seen it before, this notion will probably seem quite abstract (like a lot of formalism the first time you see it!). But it actually provides a rather precise formal match with the intuitive notion of a variable.
Added in response to the OP's comment: E.g. the polynomial ring $\mathbb C[x_1,\ldots,x_n]$ is the free commutative $\mathbb C$-algebra in the variables $x_1,\ldots,x_n$. If $A$ is any other commutative $\mathbb C$-algebra (e.g. $\mathbb C$ itself), then giving a homomorphism $\mathbb C[x_1,\ldots,x_n] \to A$ is the same as choosing $n$ elements $a_1,\ldots,a_n \in A$ ("the values of the variables") and declaring $x_1\mapsto a_1,\ldots, x_n \mapsto a_n$.
This illustrates the general principal that the free object (in some particular context) on the variables $x_1,\ldots,x_n$ is an object in which no relations are imposed between the elements $x_1,\ldots,x_n$, and so it can be mapped to any other object (of the appropriate sort) just by choosing values of the variables $x_1,\ldots,x_n$ in that object.
Variations of this point of view are how idea about variables and equations between them are implemented in contemporary algebraic geometry, for example.
In pure mathematics there is no such thing as variable, there are only constants. Consider equality $a = 5$. This supposed "variable" isn't variable at all!
On the other hand, there are those curious letters in the formulas, what do they mean you ask? Well, those are informal expressions that describe the objects we reason about, however, they do not have any precise meaning, they are not formal. It doesn't contradict that the description of the object might be perfectly fine, after all you use natural language to describe the notion of set, don't you?
Still, there is a way of formalizing this and the domain which happens to deal with such problems is called semantics, where there actually is something that is called a variable, but all the formal derivations are usually long, tedious and cumbersome. Moreover semantics is more about computer science, where the precise meaning of an expression is important for the computer that is to evaluate it (it doesn't know anything about our informal notion of variable, so we need to explain everything in the tiniest details).
In mathematics we deal with those informal expressions and "pattern-match" them with suitable cases. If you do it properly, everyone knows what do you mean (i.e. what function you want to define, etc.) so there is no need to overformalize it.
I know what I wrote looks more like a peculiar fairytale than a concrete answer, but that's the way I understand it. Hope that helps, even if only a bit ;-)
Most uses of "variable" should really be replaced by "undetermined", either in the sense "this is a fixed value, but I don't care to give it's value right now" or (in equations) "this is a fixed value, which we don't know except that it satisfies...". The first sense is what Matt E's answer describes formally.
I have a different take. I think there is quite a satisfactory definition of variable that works at least in basic algebra, calculus, and analysis (and probably other settings too). It is the same as the definition of "random variable" -- you just have to forget the "random" part. Let me explain.
First, mathematically, a variable is a function. What kind of functions deserve to be called "variables"? Often, we talk about "real" or "complex" variables. This refers to the range of the function. Variables are often valued in "numbers" (for "numeric" variables) often meaning something like a field (thus "real variables", and "complex variables"), but sometimes in set $\{0, 1\}$ for binary variables, or even in more complex mathematical object, like vector space for "vector variables" etc. etc. The domain of the function is slightly more subtle issue -- actually several related issues. They are related to 1) the ability of functions to restrict and 1) ability of functions to "pull back" (compose). The first gives us ability -- which, as Matt E points out in his answer, is critical -- to specialize the variables -- meaning restrict the functions to subdomains. The second gives us ability to transfer variables from one space (domain) to another one, which maps into it. Thus when we talk about a variable, we are at all times talking about a function, but we often change the domain on which we consider this function in the middle of the argument, without being explicit about it. (I think this is what makes the notion of variable useful, but also confusing.)
Stepping aside from formal-ish mathematics, here is how we use "variables" in mathematical modeling of systems in other subjects (physics, chemistry, economics, what have you). We are presented with a "system" (a robot, a gas, an economy, etc.) which has a set of states (modeled as some abstract set $S$). These states have numerical characteristics (joint angle of joint number 2 in radians, pressure at location P in Pascals, average price of gas in Paris in euros per liter, etc.), which are -- you guessed it -- variables. Thus a variable $v$ is a function from space of possible states $S$ to "a number". Of course there are "vector variables" (like "velocity of the robot's center of mass"), and "categorical" or "binary" ones ("what party does the president of the US belong to") etc. here as well. [This is, as I promised, the same as a "random variable" except we don't require either the set of states or the set of values to have any extra structure (no sigma algebras here) and correspondingly don't require $v$ to be "measurable".]
Often some particular collection of variables $v_1, v_2, \ldots$ is sufficient for our purposes -- it describes the state of our system well enough, so that for the purposes of the analysis that we intend to do we don't need to know anything more than values of those variables (this is akin to a "sufficient statistic" or a "faithful representation"). Then we may think of the system in question as being completely captured by these variables, and consider instead of the system its "model" aka the image $V(S)$ of the joint map $V=(v_1, v_2, \ldots)$. There are two things that happen now:
1) Any variable "on the system" $v:S\to X$ that is of interest is assumed to be a "pull back" of $w:V(S)\to X$ via $V$, that is, be of the form $v(p)=w(V(p))$ for some $w$ (this is what it means for the system to be completely captured by $V$).
and
2) Some of the variables on $V(S)$ (and hence on $S$) can be obtained by restricting to $V(S)$ a "global" variable. Here a global variable is variable defined on the "ambient space "$R$ -- the product of ranges of $v_i$s (if each $v_i$ is valued in, say, $\mathbb{R}$, and there are finitely many of them, then this $R$ is $\mathbb{R}^n$). Certainly each $v_i$ is of this form -- it is a restriction of the projection that takes $(x_1, x_2, \ldots)$ to $x_i$. These projections are now called "coordinate functions" or just "coordinates".
When we talk about variables, we sometimes refer to functions with domain $S$, sometimes with domain $V(S)$ and sometimes to the global variables with domain $R$. (Confusion between these plagues a number of discussions of calculus of variation, for example; I believe physicists sometimes refer to calculations with "global" variables as being "off shell" and ones with the restricted variables as being "on shell", but do not trust me on this point too much).
Let's consider some ramifications of this view in algebra/calculus/analytic geometry.
1) Equations: We have variables $x$, $y$ and $z$ that are global, but also restrict to variables on the system in question. When we write something like $y=x^2$ (or $x^2=1$) we are specifying a subdomain of $R$, the set of points $p$ (in $R$ or in $V(S)$ as the case may be) where $y(p)=x(p)^2$ (or $x(p)^2=1$, respectively) that is the (largest) subdomain on which $y$ and $x^2$ restrict to the same function ($x^2$ and $1$ restrict to the same function). "Solving" $x^2=1$ amounts to specifying this subdomain in a different format, usually by giving simple criterion for membership, like listing all elements ($x=\{1, -1\}$, where we implicitly use the coordinate functions, and really mean $\{p| x(p)=1 \text{ or } x(p)=-1\}$).
Aside: After a discussion of differentials (which are also functions, but again often simultaneously on tangent spaces of $R$ and on restrictions to tangent spaces of submanifolds (curves)), one can view ordinary differential equations like $dy=ydx$ in the similar way - as asking for submanifolds (curves) on tangent spaces to which the restrictions of the differentials are equal as functions.
2) Dependent and independent variables: Some $V(S)$ have the property that they are (at least locally) in bijection with (some open subset of) the images under sub-collection $(v_{i_1}, v_{i_2},\ldots)$ of (usually coordinate) variables. Then we can consider these $v_{i_j}$s as "independent variables" and all other variables as dependent variables, meaning functions (locally) obtained by pulling back functions of $v_{i_j}$s. Of course many $V(S)$ have multiple such descriptions: a circle $x^2+y^2=25$ near $(3,4)$ is locally in bijection with it's projection to the $x$ coordinate AND with it's projection to the $y$ coordinate -- we can view either $x$ OR $y$ as "independent variable" on the circle near $(3,4)$.
3) Relatedly, "a change of variables" or "change of coordinates" is then a change of which variables $v_i$ are used to represent $S$, often by composing $V$ with some "coordinate change map" from $R$ to some other range.
Etc.
