The point is that you need a meta-system MS in which to do your reasoning about the formalized language you are setting up, which could for example be FOL. You also need to work within MS to describe the formal system you wish to study, such as ZFC. In some modern logic textbooks, they will state that MS is ZFC or something like that, but that is a little misleading because there is still no way to describe ZFC without at the bottom using natural language NL. This circularity at the bottom cannot be escaped. It is not even just the problem that you use physical symbols that are slightly different with each instance of the same alphabet (as Mauro pointed out), but also there is the problem that you cannot describe the rules of any system without at some point relying on your pre-existing understanding of the concepts of "if" and "repeat"!
Psychologically, you have to use your own human mind to interpret any mathematical statement, and in doing so you have to somehow form a mental construct that you believe the author intends to convey (through whatever physical media). Who knows whether you succeeded? How can you ever know? Nevertheless, you subjectively do know whether practical mathematics seems to work empirically in enabling you to accurately predict things about the real world, so there is reason to believe that you must be somehow learning correctly what your mathematics teachers taught you.
Philosophically, one has to realize that there can be no purely mathematical reasoning about the real world, because there is simply no way to refer to any real world object without using some 'real-world hook', such as some natural language phrase! Symbols remain completely meaningless until you imbue it with meaning via some interpretation. Even classical logic itself was designed precisely to capture reasoning about our one reality. Why did we even have the concept of "true" and "false"? Obviously, because (vaguely speaking) every sufficiently precise statement about reality either holds or does not hold (and never both). A careful analysis along these lines also totally dispels the common paradoxes.