1. Why is the arbitrary union axiom in the definition of topology necessary?
2. Why is it useful? Why might we expect ("intuitively") that it should be useful?
3. What is the (historical) motivation for the axiom?
Please do not read any of the below unless you are really bored or have a lot of time to kill; the essence of my question is the above.
In what follows, I will follow Dellacherie and Meyer in denoting:
$\underset{f}{\cup}$ = finite unions
$\underset{c}{\cup}$ = countable unions
$\underset{u}{\cup}$ = uncountable unions
$\underset{a}{\cup}$ = arbitrary unions
Similar notation applies for intersections.
In what follows I will take closure under (relative) complementation for granted, simply because almost all set systems which I know of (besides $\pi$ systems) have this property, its motivation is obvious, and because it is involutory it is easily added to the axioms of a set system without changing the included sets very much.
Most spaces which are commonly studied have at most the continuity of the continuum (not that there aren't counterexamples, but at least I don't run into them in my work).
Thus, it seems we could have at most the following types of set systems for a space which has at most the cardinality of the continuum.
$\underset{f}{\cup}$, $\underset{f}{\cap}$ (Boolean) algebra
$\underset{c}{\cup}$, $\underset{f}{\cap}$/$\underset{c}{\cap}$, $\underset{f}{\cup}$ (slightly stronger than a $\lambda$ system)
$\underset{c}{\cup}$, $\underset{c}{\cup}$ $\sigma-$algebra
$\underset{u/a}{\cup}$, $\underset{c}{\cup}$/ $\underset{u/a}{\cap}$, $\underset{c}{\cup}$ ????? Related: Is there a difference between allowing only countable unions/intersections, and allowing arbitrary (possibly uncountable) unions/intersections?
$\underset{u/a}{\cup}$, $\underset{f}{\cap}$/ $\underset{u/a}{\cap}$, $\underset{f}{\cup}$ systems of open sets (topologies)/systems of closed sets resepctively
$\underset{u/a}{\cup}$, $\underset{u/a}{\cap}$ Alexandroff topologies
The motivation for the set axioms of a Boolean algebra are obvious, because they basically constitute "closure under elementary logical operations". For example, all truth functions are well-defined given a Boolean algebra, as is propositional logic.
The motivation for a $\sigma-$algebra is also fairly clear to me -- it is essentially the "countable closure/extension" of a regular algebra, and this property allows us to define countably additive set functions, and along with that obtain closure under pointwise limits of functions as well as the monotone convergence theorem, the two foundations on which integration theory is based.
It is also clear that such additivity could not be extended to the cardinality of the continuum, since the uncountable sum of any non-zero numbers is infinite.
But why do open/closed set systems turn out to be useful? How are these uncountable axioms applied fruitfully in topology?
As far as I am aware, for example with stochastic processes, we usually rely on some separability/first/second-countability properties in order to be able to apply most topologies fruitfully -- i.e. some reduction of the "uncountable" axiom to an essentially countable property. So why not study only metric spaces?
Moreover, the study of limits is only defined/sensible once we have the T1 and T2 separation axioms, so the claim that we study set systems of type 5. in order to study limits seems unjustified to me; at the very least we need that in combination with some separation axioms.
Also, why do we study set systems of type 5., but not of types 4. or 6.? What is the essential property that type 5. set systems capture (in the way that type 1. and type 3. systems capture)?
The best discussion of this topic which I have found so far: https://mathoverflow.net/questions/19152/why-is-a-topology-made-up-of-open-sets
I agree with the OP of that question however, that the top-voted answer is more of a justification for the notion of metric space (which are always separable in some sense due to the density of the rationals in the reals).
The discussion of how notions of limit really seem to be more of a consequence of a notion of "Hausdorfness" is here: Why not just study the consequences of Hausdorff axiom? What do statements like, "The arbitrary union of open sets is open," gain us? I would also like to point out that the axiomatization of topology in terms of neighborhood systems, plus the Hausdorff axiom, seems to be more relevant to understanding the concept of "limit" than the axioms of a general topological space. And again, I don't see why we would need uncountably many neighborhoods of a given point to define it as the limit of something, since countably many neighborhoods always suffices (at least in the metric space setting).
Motivation:
The motivation for this question came from some rumination about stochastic processes. Namely, the reason why their study can be so difficult is that we are confined to using set systems of type 3., which are only countably closed, whereas we are studying uncountably many random variables.
In order to get any degree of meaningful results, we therefore rely on having some degree of "separability/regularity" properties that essentially allow us to reduce to the countable case where it counts (for instance with regards to the existence of regular conditional distributions).
So as frustrating as it may be trying to study uncountable objects with set systems of type 3., it is clear to me why we can not replace such set systems with any of type 4, 5, or 6, again so that integration and probability are tractable.
So then I thought "oh so we must use topologies because we need uncountability for some reason". But then I thought about how almost all topologies which I am aware of/use are in some way "separable" or "first/second-countable", which basically constitutes reducing back to the countable case, the same way we do with stochastic processes. I.e. there is no fundamental gain in generality, since the only cases of uncountable objects we can/do study are those special cases which can be reduced back to the countable case in some meaningful way.
I'm really at a loss as to why an uncountable closure axiom would be at all useful, and why we would want only finite closure for the complementary operation, rather than some type of infinite closure.
I feel like it has to have something to do with the distinction between "discrete" and "continuous" spaces, which leads us back again to the study of stochastic processes, and the distinction between "discrete" and "continuous" time.
Namely, for a set with countable cardinality, like the natural numbers, the power set coincides with the reasonable choices for sigma algebra and topology.
In other words, both concepts are clearly only needed when moving to the uncountable case, where they don't necessarily coincide with the power set, i.e. we can gain a concept of "continuousness", rather than the "discreteness" which we are necessarily confined to with set systems of countable sets. Again, the purpose of $\sigma-$algebra in this case is clear to me, since it is actually only closed under countable set operations, but now I really don't understand why we are able to use the uncountable set axiom profitably in open/closed set systems, especially since in all cases which I am aware of we have some sort of "separability" which reduces the concept of topology to a countable one. So why not define the concept which we actually use, which is countable, not uncountable in nature?