I think this question can be taken into several different directions. If you substitute "interesting" or simply "nonarbitrary" for "meaningful" (as Nate Eldredge and Pete L. Clark did), at least three criteria come to mind: The first is simplicity; this can be made precise in various formal systems, e.g. the natural numbers are in some sense the simplest infinite set, etc. Second, if some object appears naturally while researching something else, that can suddenly make it significantly more interesting. Actually, I think such cross connections are often a boost for both sides. So for example, if $x+2y-1016z$ happened to be, say, the only polynomial with integer coefficients satisfying some non-obvious property, it wouldn't be arbitrary any more -- perhaps even if the subject where that property came up didn't get a lot of attention before. The third factor is how many nontrivial theorems can be proved about an object. I suppose this only makes objects more interesting if they are already interesting for other reasons, though.
The issue of complexity is especially subtle, even if you can probably find a good metric. For example, the definition of a Turing machine is rather lengthy with quite a few arbitrary decisions. What makes Turing machines interesting anyway is the Church-Turing thesis. In other words, the truly interesting property is Turing-completeness, and the truly interesting object is the class of all Turing-complete machines, while the Turing machine itself is just an arbitrary representative of that class. The special gem here is that there does not seem to be any simple/natural/nonarbitrary/interesting way to formally define that class. This suggests that one should distinguish between mathematical objects, which may be interesting, and their descriptions, which may be arbitrary nevertheless.
This brings me to another possible interpretation of the words "meaningful" and "arbitrary." Axiomatic set theory (e.g. ZFC) can be said to trade "arbitrariness" (is that a word?) for complexity. Take the Kuratowski definition of an ordered pair, for example: It is not just asymmetric, it even implies that (under the usual definition of natural numbers) the pair $(0,1)$ is a proper subset (yes, subset!) of the number $3$. You can replace $3$ with any larger number, but not with $2$...
Here, one might argue that the question whether something is a subset of the number $3$ is not meaningful. This may not be exactly what you had in mind, but I consider this concept of meaningfulness quite important, and it can be made precise in various ways. Type theory provides a very satisfactory formalization, but since it is an alternative to set theory, it leaves open the question of why we consider a particular statement meaningful or meaningless in (set-theoretic) mathematical practice. This question has been researched on philosophical grounds, and one can also try to find a formal system that only permits meaningful statements. (I have been working on this.)
A weaker version of "meaningful" vs. "arbitrary" comes from mathematical incompleteness. The independence of e.g. the continuum hypothesis from ZFC gives objects like $\aleph_1$ a more "vague" feel, thus some people consider such questions or objects less meaningful than others. But there are also very concrete independent statements, most famously (in this case) the question whether ZFC itself is consistent. A key difference is that the continuum hypothesis does not have any arithmetic consequences.
To conclude, if your question is interpreted sufficiently broadly, a lot of well-researched foundational topics come to mind. In any case, there cannot really be a single answer.