These two theories are known to be first-order theories. And the definition of the first-order logic typically involves something like "set of symbols $a_1, a_2, a_3...$", which already includes set and numbers. And also the proofs by induction are also always accepted without any underlying rigorous reasoning.
So, ZFC and Formal arithmetic and their corresponding intuitive analogues are basically two completely different things, one built using the other, not the different ways to represent the same object.
Since the answer to my question is most likely just "yes", I'll expand it a little bit:
Is there any standard conventional "list of primitive notions", such as collection(as informal set), number etc. that all the logic is built upon?