Not really an answer, just remarks (but you have not really asked a question either):
I found it enlightening when I read (in my case, for the first time in Frege's Grundlagen der Arithmetik) that numbers are not something that applies to physical reality, but to concepts. You are already working with abstractions when you get to apply numbers. Simplified example: There is no such thing as "three apples". There's some visuals, smells, sounds. But your mind at some stage has developed the concept "apple", i.e. the idea that there is something that can be called apple, and something that cannot be called apple (e.g. a pear, or cigarette smoke, or a memory of your grandmother); and of the former concept, there happen to be three on the table.
In a Kantian view, numbers are not part of the "Ding an sich", but part of the way our reason orders reality. As shown in Q the Platypus' answer, numbers come up once one has the concepts of "collections of objects" and "bijections between such collections". Neither of these concepts are a part of physical reality; they are ways how our mind makes sense of the world.
As an aside, it's maybe interesting to think about how e.g. concepts of continuous motion, or measurement, actually come earlier in childhood development than the natural numbers. In a way, some properties of the continuum (the real number line) are even more basic and intuitive than counting objects. So you might as well ask, what's the relation between the idea that something is "bigger" or "smaller" than something else to physical reality, or the idea that if two things are not the exact same size, then one is bigger and the other is smaller, and that then there is always something whose size is between them ... is that grounded in physical reality?
Addendum: In the comments, user Shahab links to the question https://philosophy.stackexchange.com/q/49807/40478 where several comments and parts of answers raise points maybe better than I could do it. In particular, I want to quote from one answer by user "Ben - Reinstate Monica":
Remember that numbers (of any kind) are an abstraction that is used to describe concrete aspects of reality. To say that a mathematical object "is part of reality" is false in the concrete sense, but it can be true in the metaphorical sense that aspects of reality are accurately described by those abstractions. In the case of complex numbers, part of the confusion here comes from incorrect understanding of what they are ("but they're imaginary", etc.), which leads people to set them apart from other types of numbers, and imagine they their "existence" is somehow stranger than the "existence" of the real numbers, rational numbers, etc. [and here I add: natural numbers!]
and in particular this comment by user Dan Bryant:
As an aside, I challenge the implied presupposition that natural numbers of mangoes are inherently physical. Natural counting is certainly intuitive, but it presupposes that we can clearly and unambiguously identify mangoes, separating them into individual objects to count. I suggest that this is non-trivial and only appears obvious by virtue of the way our cognition and perception function.
which, if I'm not mistaken, is exactly what I tried to convey with my "apples" above.