When you start to consider more general ideas than measuring geometric shapes and counting elements of sets, you start using more general sorts of objects to quantify those ideas. Or sometimes you consider interesting algebraic structures the are analogous to familiar ones, but with some other sort of thing fitting into the role where "numbers" fit into the familiar ones.
I'm of the opinion that it's reasonable to call such things numbers. However, I don't think I would ever say that out loud (other than in an opinion piece), since I would expect listeners to be confused by my usage of the word, except in cases where the word "number" is traditionally used (e.g. we say "ordinal number" versus "well-order type").
Commonly in such situations there are other words available: for example, "scalar".