The notion of a set being a "collection of objects" is admittedly rather vague, but it helps to think of a set as being an abstract mathematical concept. Whether or not a set is infinite says nothing about our ability to "fit all of the elements into it", as if you are fitting real-life objects into a box. Instead, the term infinite has a precise mathematical definition. There are a number of equivalent ways to say precisely what an infinite set is. Here is one possible definition:
- If $A$ and $B$ are sets, we say that $A$ is a subset of $B$, written as $A\subseteq B$, if every element of $A$ is an element of $B$. If, in addition, $A\neq B$, we say that $A$ is a proper subset of $B$.
- If $A$ and $B$ are sets, we say that $A$ and $B$ are equivalent if there is a function $f:A\to B$ which is one-to-one and whose image is equal to $B$.
- A set $A$ is said to be infinite if it has a proper subset $B$ which it is equivalent to. Otherwise, we say that $A$ is finite.
Notice that according to this definition, infinite is the basic concept, whereas finite simply means not infinite. If you find this unsatisfying, then there are other equivalent definitions where finiteness is the basic concept.1 See, for instance, Halmos's book Naive Set Theory.
In spite of the apparent precision in the above definition, you might protest that the idea of a set being a "collection of objects" is still rather nebulous, and you are right. This is ultimately one of the motivations for axiomatic set theory. In axiomatic set theory, all of our reasoning is based on a list of axioms intended to describe the properties of sets (more specifically, the pure well-founded sets, as motivated by the cumulative hierarchy). The most common axiom system is known as Zermelo-Fraenkel set theory ($\mathsf{ZFC}$). For instance, one of the axioms, known as the Axiom of Extensionality, states that
$$
\forall A\forall B(\forall z(z\in A\Leftrightarrow z\in B)\Leftrightarrow A=B) \, .
$$
We usually think of this axiom as "saying" that two sets are equal if and only if they have the same elements. However, this is just the informal interpretation that we assign to the above string of symbols. Notice that the word "set" and the phrase "collection of objects" do not feature at all in the above axiom, and there is nothing that compels us to think of $\in$ as being the "true membership relation" (if such a thing even exists). In axiomatic set theory, $\in$ is a primitive (undefined) symbol. We say how $\in$ behaves, through our axioms, but we never make any ontological claims about what $\in$ "is".
In fact, there is not a single proof in $\mathsf{ZFC}$ that ever uses the fact that we interpret the variables to be ranging over the "universe of sets", and $\in$ as denoting set membership.2 Instead, everything follows precisely from the axioms, according to a fixed collection of deductive rules known as the inference rules. Moreover, if a proof in set theory is written out in full using the inference rules, then its correctness can be verified mechanically by a computer that does not "know" anything about sets or set membership.3
This means that we have an agreed standard of proof in set theory, without ever having to worry about the thorny question of what a set actually "is". That question becomes a philosophical one, and though it is fascinating to contemplate, you don't actually need to ponder it to do real mathematics.
1Technically, the definition I have given is that of a Dedekind-infinite set. Set theorists tend to prefer the definition that an infinite set is a set which is equivalent to a proper initial segment of $\mathbb \omega$. The reason is that, to prove that a set is Dedekind-infinite if and only if it is infinite, we need to use (a weak version of) the axiom of choice, and some set theorists are interested in doing set theory without the axiom of choice. It is consistent with $\mathsf{ZF}$ that there is an infinite set which is Dedekind-finite, i.e. not Dedekind-infinite.
2Hilbert's quote about "tables, chairs, and beer mugs" seems apropos here.
3An example of a proof being written out "in full" can be found here. This kind of proof is known as a formal proof, to distinguish from the informal proofs that we write out when we are doing ordinary mathematics. There is some further discussion of the distinction in this post.
