I think the idea is that lines are sets of points. The length of a line is equal to the cardinality of the set of the points that comprise it.
Said otherwise, let line segment $P_{1}P_{3}$ designate the set $\langle P_1,P_2,P_3 \rangle$. The line segment is 3 units long.
In this way, assigning actual measures, such as inches, to units isn't any different than assigning actual things to numbers. For example, $2$ apples.
Math is abstract. Abstraction involves leaving things out (alternatively, concretion involves adding things in; however you prefer). Leaving out physical things allows us to generalize, which is a reason that math is so broadly applicable.