Although mathematicians go nuts over this stuff, abusing notation and working with symbols in a way that relies on intuition is often critical to a physicist and useful to a mathematician who wants to build deeper understanding and intuition. We should all be grateful that not everyone is paralyzed by rigor-mortis because scientific discovery benefits from the audacity to break formal rules.
A mathematician would say that for a parameterized curve $\gamma(t) = (x(t), y(t))$, the arc length along the curve between parameter values $t_a$ and $t_b$ is
$$
\int_{t_a}^{t_b} |\dot\gamma(t)|\, dt = \int_{t_a}^{t_b} \sqrt{\dot x(t)^2 + \dot y(t)^2}\, dt
$$
This definition is motivated by thinking about little pieces of time $dt$ and the fact that $|\dot\gamma(t)|$ is the speed at time $t$. For little pieces of time, the speed times time will give a little straight line piece of distance traveled, and then one simply adds up all the distances of these little pieces. In other words, there is a very nice physical motivation for this definition of arc length along a parameterized curve.
Now as a special case, suppose that there is a parameterization of the curve whose arc length we're trying to compute taking the following form:
$$
\alpha(s) = (s, y(s)).
$$
Since the $x$-coordinate function is simply the identity function, we might as well call this parameter $x$ (it's a dummy variable anyway), in which case we get
$$
\alpha(x) = (x,y(x)).
$$
Now plug this into the original arc length definition to obtain
$$
\int_{x_a}^{x_b} |\alpha'(x)|dx
= \int_{x_a}^{x_b}\sqrt{1 + y'(x)^2}\, dx.
$$
This is precisely the formula written down by the physicist.
Comment on the Edit. Treating derivatives as difference quotients of small quantities often works because, well, that really is what a derivative is doing. Look at the definition of the derivative as a limit of a difference quotient. If $\Delta x$ is small, then replacing the derivative by the difference quotient won't generally incur a large error, so it's not always such a bad way to look at things. This should, of course, be taken with a grain of salt when you want to rigorously clean everything up in the end, but it's often unproductive to tie your hands and not think about these things intuitively, especially when you're first learning them.
I enjoy and appreciate rigor as much as any other respectable citizen off the street, but I've also learned to appreciate that working loosely with mathematical quantities can often lead to great intuition and insight. Take, for example, path integrals in physics. No one really knows how to define these beasts in a way that would satisfy a modern mathematician (especially path integrals in quantum field theory), but nonetheless physicists' formal manipulations have led to some of the most accurately predicted measurements in human history.