The simplest infinite set in structure is definitely the set of all Zermelo naturals, i.e. the set of all iterated singletons of the empty set. The set of all Von Neumann naturals is not the simplest in structure, but if one requires a definition of the ordinals that suits many of them being infinite, then its here were Von Neumann's would trip in.
Its funny to see that even for that purpose Von Neumman ordinals are still not the simplest in structure! The simplest would be sets iteratively formed by starting from the empty set then applying the singleton operator at successive stages; and at limit stages taking the set of all sets formed at prior stages.
What really von Neumann's enjoy is having the SIMPLEST definition of the strict smaller than relation $<$ on ordinals, which is the most important relation about ordinals. This would be set membership itself on ordinals! By comparison, with the other approach, this would be defined as: .. is an element of the transitive closure of.., which is more complex.
Its easier to also well order ordinals horitzontally (i.e., using Von Neumann's) than just doing it vertically (like the above recursively formed ones)
Of course at the finite level, its easier to define the relation successor $\mathcal S$ using the iterative singletons method, since it would be simply $\in$ on ordinals, and $\mathcal S$ is indeed the pivotal relation at that level, but that advantage stops on the shores of the first transfinite ordinal $\omega$.
Using the infinite kind of iterative singleton formation depicted above, then the relation $\in$ on ordinals would define the following relation on ordinals $$ y=\mathcal S(x) \lor [y \text{ is a limit } \land x < y]$$
The point is that this relation doesn't seem to be pivotal in thinking about ordinals, not at any rate comparable to the relation $<$.
Von Neumann's being transitive is not the basic merit of them, we can have a definition of ordinals that makes them transitive and yet they are not the von Neumann's, for example take ordinals defined recursively starting from the empty set, then we define successors of limit stages (Zero included) by the union of the limit stage and the singleton of it, then we define successor stages of successor stages as: $$\mathcal S(x)= x \cup \{\{max(x)\}\}$$, and take the union of all prior ordinals at limit stages.
This is less in structure than the von Neumann's, the ordinals defined are transitive, we can define $\leq$ relation on ordinals in a nice manner as the $\subseteq$ relation on ordinals, of course $<$ on ordinals would be defined after proper subset relation on ordinals, however this is more complex than defineing $<$ as simply $\in$ on ordinals.
So the balance goes to Von Neumann's at the infinite realm!