First, a disclaimer: I'm of mixed opinions regarding nonstandard analysis in early curricula. Below I'm focusing mainly on the objections to NSA there, as that's what the question is asking about; my actual opinion is more balanced. (See the end of this answer.)
Second, a bit of context. There has been some research on the effectiveness of NSA in the curriculum; my understanding is that, while there has not been much, the results have been generally positive. In particular, the study I want to have in mind for what follows is the one summarized in this article by Kathleen Sullivan, which I will quote from a bit.
Here are some reasons I think NSA has not caught on, curriculum-wise:
- Non-pervasiveness at the higher level. Silly version first: if someone learns calculus via NSA, and then goes into mathematics, they will find themselves having to re-learn calculus to a certain extent, since NSA is not the language in which most analysis, at the research level, is currently conducted. OK, that's silly because it only applies to a tiny handful of students, and these are the students who are most equipped to learn multiple foundations; but we also see the same problem in higher-level math courses! Continuity in topology is now much more mysterious, since general topological spaces have no "hyperreal versions" (or rather, some do, but many don't); and classes building on analysis (complex analysis, differential geometry, etc.) would also need to be altered. So to do this properly, we wouldn't just need to change one class - large portions of the math curriculum would be affected. (Incidentally, note also that this would heavily affect transfer students, as well as students who have already seen some calculus via $\epsilon-\delta$ in high school.)
This issue is partly acknowledged by the Sullivan article:
On the other hand, some uncertainty was voiced on the
question of how well students who want to study more analysis will be able to make the transition
from an experimental class to a traditional course. Conversations with students at the University of
Wisconsin, who had been in nonstandard calculus classes, suggest that the attitude of the instructor in
the standard class may be the crucial factor.
There is also the dual issue of instructor unfamiliarity, which interacts with the above objection in some obvious ways.
Of course, there's the issue that NSA is useful and used at the research level! But it's still a very much minority tool. So my objection is still, I think, a serious one.
- Strange number system. The real numbers form a nice number system - in particular, every real is easily distinguishable from every other real. Formally, given reals $r\not=s$, there is a first-order formula in the language of fields true of $r$ but not $s$, and vice-versa (look at a rational between $r$ and $s$ . . .). By contrast, any hyperreal field has lots of nontrivial automorphisms - that is, there will be lots of hyperreals which are indistinguishable from each other. This is a very weird property for a number system to have, and in my opinion is going to make the hyperreals a difficult concept for the students to grasp - in particular, the question "what is a hyperreal number?" is going to have a much less satisfying answer than the question "what is a real number?", and the student who asks for an example of an infinite/infinitesimal number may reasonably be very disappointed with the answer (I certainly was)! In particular, the more Platonistic students (and in my experience, a large number of students are what might be called "Platonist-by-default" - they've never seriously considered the question of mathematical existence, which to be fair is quite reasonable) will have good grounds for objecting to this approach.
Relatedly, there's the issue that many/most mathematicians - including me! - view the specific structure $\mathbb{R}$ as interesting and worthy of study, moreso than any specific hyperreal field, or indeed the general class of hyperreal fields. Which is not to say that NSA doesn't shed light on $\mathbb{R}$ - obviously it does, that's the point - but rather that the $\epsilon-\delta$ framework is a beautiful construction, and interesting in its own right. Of course, the fact that I consider it beautiful is revealing of my own context as mathematician vs. student, so I'm not bringing this up as a main objection; but I think it's worth keeping in mind that this may be an objection the instructors have. And this may also be an issue for higher-level students; see below.
- Difficulty of rigorous formulation. Consider the honors section. It's easy to ground standard analysis in an "obviously consistent" theory - namely, develop the theory of the reals via Cauchy sequences, from basic set theory and the field axioms. Note my scare quotes around 'obviously consistent' - of course, I recognize that the consistency question for analysis is much more subtle than that! But my point is that an honors student seeing this development will with high probability be convinced that calculus is consistent. However, a strong calculus student may reasonably not be convinced of the consistency of NSA, specifically the question of why we believe we can postulate a hyperreal field! This is where the logical complexities of NSA become a problem - there is no good answer to this question which is accessible to the student at this time. Of course, this only applies to students who (a) are curious enough to challenge the consistency assumptions, and (b) are strong enough that they could follow the development of standard analysis; but there are lots of these students, so it would be odd to ignore them.
Alright, let's get a rebuttal. Note that all the objections above were focused on the higher-level students, to varying extents. This is because as long as the students do not continue on to higher math classes, and as long as questions about mathematical reality, consistency, and the like are not brought up, NSA seems to have a strong advantage. Again, from the Sullivan article:
The [NSA] group was also given the edge regarding
the ease with which they were able to learn the basic concepts. One instructor commented that, "When
my most recent class were presented with the epsilon-delta definition of limit, they were outraged by its
obscurity compared to what they had learned."
Let me finish by putting some "skin in the game" and stating my own opinion for all to see:
I think NSA is undoubtedly the better framework for the student who intends to learn calculus, and not proceed further in mathematics. And this is a large number of students, and a reasonable position to hold on their part. I think the situation is much more mixed - though by no means necessarily fully, or even mostly, negative! - with regard to students who intend to continue on to higher-level math classes, and potentially a serious detriment to those who intend to go into research mathematics (also, to be fair, potentially a serious boon - my point is that I'm worried, not that I'm certain of negative outcomes). I also think that it would meet with resistance on the part of many/most instructors; while this isn't an "ideal" problem, it's a reality that has to be acknowledged. And I am doubtful that it would vanish over time, given the difference in roles of standard analysis vs. nonstandard analysis in research mathematics (although it probably would diminish).
Ultimately, I want more data. For one thing, the study analyzed in the Sullivan article was quite small; for another, it didn't have anything to say about the students' future math classes. I could be convinced that NSA is the superior method, but currently I am not.
And of course the above is not in any way meant to obviate my many and strong objections to all sorts of things surrounding calculus in the curriculum; but that's a very different issue, so I'll stop here.