"I can't think of any set that doesn't have an order"
This attests to how fundamental order seems to be, and why it's one of the foundational axioms of modern set theory. The well-ordering theorem (equivalent to the Axiom of Choice), apart of ZFC, is simply that; an axiom. An assertion that for any set there exists a relation (which could be more of an abstract relation than your typical $>$ or $<$) that well-orders that set. There's nothing stopping you from using a set theory without such axiom, as many set theorists do.
The answers to this question goes over the difference between total order, which is what Rudin is referring to, and well order.
The set of complex numbers is a good example of set with undefined order. However, you can order them given a more abstract relation.
It's important to note that no set naturally comes equipped with an ordering relation. The relation is something that is usually defined after a set is defined. For example, how do you order the set $\{a,7,\uparrow\}$? There's no "right" way, but you can create a well-defined relation that orders such set, just as was done for the real numbers and etc.