The issue is you're comparing a physical encoding of the abstract concept of addition to the mathematical description of it.
Here's an example which will really mess your intuition of binary operation. Suppose I gave you a basket of balls, and I myself have a basket of some number of balls.
If I ask add three balls in my basket, then without looking inside my basket you can simply throw balls into my basket.
But what if I ask multiply the number of balls in my basket by three? Then you can not do this unless you have some way of knowing how many balls there are in my basket.
Now does this mean multiplication is some way beyond a binary operation ( in math sense)? Well... no. Its just that this physical realization of the addition comes with more properties than the mathematical description of it.
The thing is mathematical objects and descriptions of them don't neccesarily need to conform to our intuition or a particular model in real life.
We don't go from the physical world to mathematics but rather try to explain mathematics as a thing itself using logic. That's the whole idea of axiomatic systems and proofs. One more point is that this is also a hint that the teaching mathematics by extrapolating physical intuition of a small set of scenario to a mathematical description which works anywhere is problematic.