We all know that if $(X_1,\tau_1)$ and $(X_2,\tau_2)$ are topological spaces, then a continuous function $f:X_1\to X_2$ is defined to satisfy the requirement that the inverse image of any open subset of $X_2$ is an open subset of $X_1$. That is, $f^{-1}(A) \in \tau_1$ for $A \in \tau_2$. Similarly, if $(X_1,\mathcal{A}_1)$ and $(X_2,\mathcal{A}_2)$ are measure spaces, then a measurable function $f:X \to Y$ is defined to satisfy the requirement that the inverse image of any measurable subset of $X_2$ is a measurable subset of $X_1$. That is, $f^{-1}(A) \in \mathcal{A}_1$ for $A \in \mathcal{A}_2$.
While I understand why morphisms in these specific categories are defined this way, in the abstract I assumed that we choose inverse image because inverse image is well-behaved under unions and intersections. That is, lets say $X$ and $Y$ are sets, and $T_X \subseteq \mathcal{P}(X)$ and $T_Y \subseteq \mathcal{P}(Y)$ are some prescribed subset families we use to abstract a category (like open sets for topology or $\sigma$-algebra for measures). If $f:X \to Y$ is a function of sets, then $f(A \cap B) \subseteq f(A) \cap f(B)$, but with inverse image $f^{-1}(C \cap D)=f^{-1}(C) \cap f^{-1}(D)$. So while we have two candidates $f:\mathcal{P}(X) \to \mathcal{P}(Y)$ and $f^{-1}:\mathcal{P}(Y) \to \mathcal{P}(X)$ we could use to define the "right" notion of morphisms, the latter is better behaved than the former, since $T_X$ and $T_Y$ are closed under unions and intersections.
But what exactly is going here? For instance, defining continuous functions to be open maps fails in capturing our intuition of continuity, sure, but abstractly, what is wrong with it?
I am asking because I encountered another category: simplicial complexes. A simplicial complex too can be described abstractly as a set $V$ (of vertices), and a certain collection $S$ of finite subsets of $V$ (the simplices). However, we don't require $S$ to be closed under unions now. What we do prescribe is that $S$ contains all singletons and is downward closed under subsets. In this category, a morphism of simplicial complexes $(V_1,S_1)$ to $(V_2,S_2)$ is a map $f:V_1 \to V_2$ such that for any $A \in S_1$, $f(A) \in S_2$.
So here too we define the category in terms of a collection of subsets satisfying certain conditions, but for defining morphisms, we don't use inverse image.
My question is:
If we want to define morphisms on a some category that is defined by prescribing subset collections, when do we use image and when do we use preimage? What is the "right" or "natural" notion and how should we abstractly understand this?