This question straddles philosophy and mathematics. Or perhaps I should say, it involves both philosophical and technical questions about mathematics. Also it is ripe with history.
On the one hand, informal reasoning about syntax and proofs can be carried out formally, via coding, inside Peano arithmetic or even weaker systems. This insight was of course critical to Gödel's incompleteness results.
On the other hand, to understand this formalism, you must already possess a certain base level of "mathematical intuition", including some notion of what the natural numbers are, and some feeling for "why" induction is "true". (Scare quotes to indicate that we are wading into philosophical waters.) No matter how many times I explain proof by induction to my golden retriever, he just doesn't get it.
What one concludes from this... well, there doesn't appear to be a "one size fits all" answer, good for all practicioners of mathematics. Famously, intuitionists like Poincaré and Brouwer felt that the base intuitive layer was fundamental, and one should not make too much of the formal approach. I feel sure that if they were alive today, they would be skeptical of things like large cardinal axioms.
Gödel himself was a Platonist, and believed that there is a real, unique mathematical universe "out there". A formal system like PA or ZFC is our human attempt to capture some "facts" about this universe. He was even comfortable claiming that $2^{\aleph_0}$ has a "true value", and hoped (even after Cohen's independence results) that one might discover "true axioms" of set theory that would determine this value.
Finally, the formalists supposedly believe that mathematics is a game with symbols. Perhaps this elevates formal proofs with induction to a higher level than informal reasoning. I say "perhaps", because there are many flavors of formalism. Hilbert himself was not a formalist in this sense; his formalist gambit was an attempt to beat back the claims of the intuitionists that certain traditional forms of reasoning were illegitimate.
Perhaps I shouldn't have said "finally". While formalism, intuitionism, and Platonism are often listed as the triumvirate of philosophies of mathematics, there are others. The combinatorialist Gian-Carlo Rota was an advocate of something called phenomenalism; you will find many essays by him on this topic in the collection Indiscrete Thoughts. Reuben Hirsh and others claim that mathematics is a shared social construct, something like legal systems. I mention these just to indicate the wide landscape of viewpoints. (Personally I don't understand Rota's phenomenalism, and I don't buy the social construct theory.)
John Baez and I discussed some of these matters in this thread. For example, consider this quote from Harvey Friedman, in "Philosophical Problems in Logic":
I have seen some ultrafinitists go so far as to challenge the existence of $2^{100}$ as a natural number, in the sense of there being a series of “points” of that length.
It seems unlikely that the ultrafinitists would accept induction in the usual sense. (You'll find the full quote here.)