Does the inverse of an infinitesimal give an infinite number? That depends on what you mean by "infinite number". The term is used in many different senses in mathematics. For the purpose of doing calculus, it is helpful to use a more specific term that is also commonly found in the literature: "unlimited". An infinitesimal is smaller in absolute value than every positive standard real number, and an unlimited number (the inverse of an infinitesimal) is greater in absolute value than every standard real number.
Abraham Robinson was the first to pioneer a usable approach to infinitesimal analysis based on infinitesimals in the 1960s. His approach could be described as a model-theoretic one, and involves extending the real numbers $\mathbb R$ to an ordered field of hyperreals ${}^\ast\mathbb R$.
About a decade later, an alternative approach to analysis with infinitesimals was developed by Hrbacek and Nelson that could be called an axiomatic approach. Here the usual language of set theory based on the membership relation $\in$ is extended to a richer language incorporating a new one-place relation, or predicate, "standard", together with axioms governing the interaction of the new predicate with the usual Zermelo-Fraenkel axioms. Here it is important to keep in mind that the "axiomatic approach" does not amount to a brave new world of axioms beyond Zermelo-Fraenkel; Hrbacek and Nelson proved conservativity results that show that the new systems are conservative over ZFC. Here one defines $\mathbb N$, as usual, as the smallest inductive set; $\mathbb Z$ by adding negatives; $\mathbb Q$ as its field of fractions; and the real line $\mathbb R$ as the set of Dedekind cuts on $\mathbb Q$.
In the axiomatic approach, one finds infinitesimals and unlimited numbers within the real line itself (rather than an extension thereof). The definitions are exactly the ones given above.
Then infinitesimals can be defined in a single sentence.
I add a historical note based on an exchange of comments. The following is a summary of our recent publication. Leibniz had a distinction between assignable and inassignable numbers, which was only properly understood with Robinson's theory when it was formalized as the distinction between standard and nonstandard numbers. As late as the middle of the 19th century, infinitesimals were still in routine use by the greats such as Cauchy, Poisson, and Coriolis. The rigorisation of analysis achieved in the next generation of mathematicians was of course a great accomplishment, but in one significant way it was also a failure. Namely, the formalisation of the foundations that was eventually developed failed to include a formalisation of infinitesimals, which until then were the bread and butter of analysis. At any rate, the ZFC foundations were specifically designed to exclude infinitesimals, which were routinely considered contradictory. It is to be noted that mathematical validity is not measured by how "new" a theory is. The axiomatic foundations for infinitesimal analysis are a more faithful formalisation of analysis than those provided by the traditional non-infinitesimal axiomatisations. Recently we developed an axiomatisation whose three additional axioms correspond to three principles already found in Leibniz; see this article. They are no more elaborate than the traditional foundations.
