Amusingly, one of the answers to the question you ask is that elementary calculus rigorously defines infinitesimals.
How does it do so? Via the notion of differential. The problem you're struggling with is almost backwards; the standard, traditional track is:
- Define the notion of derivative
- Use multivariable derivatives to define the notions of (tangent) vector and differential
- Conceptualize a notion of an "infinitesimal" neighborhood of a point
To elaborate on that last point, you're supposed to envision the points of the infinitesimal neighborhood to be enumerated by tangent vectors — the intuitive idea is that you take an "infinitesimal" step proportional to the tangent vector. Differentials are the functions on the infinitesimal neighborhood.
But this conceptualization is not trying to define anything new — it is merely a way of thinking about calculus. (albeit a very useful one!)
But the second point is bog standard. For example, in multivariable calculus, one incarnation of these notions is
- Tangent vectors to points in $\mathbb{R}^n$ are $n \times 1$ column vectors — the sort of thing you get when differentiating a vector function of one variable
- Differentials at points in $\mathbb{R}^n$ are $1 \times n$ row vectors — the sort of thing you get when differentiating a scalar function of $n$ variables
Both the subjects of differential geometry and algebraic geometry treat this sort of thing much more explicitly and more in-depth.