DISCLAIMER: Not sure how good or professional this explanation is going to actually be.For all I know I'm totally wrong. All of what I have to say is just my fundamental belief based on what I can conclude from research I have done going into a senior thesis I am working on, as well as general curiosity from trying to answer myself THE VERY QUESTION you have asked. Hopefully this helps.
A formal mathematical statement like "every bounded set has a least upper bound in $\mathbb{R}$" are just words that syntactically behave like a classical predicate logic. And our proofs that we write down on paper and accept in academia follow the "inference rules" that we all know and love. Why? No other reason than "this is how mathematicians talk." Why do mathematicians talk this way? Because it makes sense. We can place rigor in a sensible way that doesn't run us into errors that tear the mathematical system apart.
The point being: These "mathematical objects" you speak of are nothing more than an abstraction. And the whole purpose behind abstractions are to cleverly model the things in a "logical" way so that it "behaves" the way we want it to. For example, set theory came up historically as one of the first modernly recognized "fundamental" systems of classification. When we started seeing (through Cantor) that not all "infinite classifications" (such as $\mathbb{N}$ and $\mathbb{R}$) were equal--despite what great minds like Galileo believed--it became necessity to "keep a tab" on the "elements" belonging to our classifications. And thus set theory was born, and dandy things like "cardinality" with it!
When one says set theory is a "foundation of mathematics", one really doesn't actually mean "THE first principle of mathematics" (although it is certainly easy to think that...only to be initially disappointed once proven otherwise, but I promise you there is a beauty in this "incompleteness" from within). The real grounbraking-ness in set theory was its ability as an abstraction to adequately model things like number systems, algebraic structures, geometric structures, and so on. But set theory is nowhere near close to a "first principle", particularly when you include what logicians would love to call "metamath": Abstractions of abstractions of things that resemble "everyday arithmetic".
For that, you would need something like "the category of all sets" which is "a proper class" as opposed to a set, axiomised using the Von Neumann–Bernays–Gödel (NBG) axioms. And it doesn't stop there. "The category of all classes" is not a class, and we essentially need a replica of the (NBG) axioms for whatever we want to call the "larger object" that contains all classes. And we can go ON forever.
But that's not a problem. Because like I said one doesn't need an actual "first principle" to do math "correctly". All we need is "well-behaved" abstraction to model whatever it is we want to do and then just do it! Whether that abstraction be set theory, first order predicate logic, type theory, universal algebra, etc. Furthermore sometimes it brings us amazing theoretical insight to relate one such abstraction to another (such as real analysis with topology, a synergy which in fact was very intentional by the inventors of topology). But that doesn't mean "one comes exclusively from another" necessarily. It just means "you can use one to explain another," which is not to say there isn't another way. For example, category theory is commonly used to explain "natural deductions" in a language, but my project intends to take a more graph theoretic approach.
Hopefully that makes at least some sense.
