ZFCfin, ZFC with infinity replaced by its negation is biinterpretable with PA, Peano Arithmetic. This result goes back to Ackermann,
- W. Ackermann. Die Widerspruchsfreiheit der allgemeinen Mengenlehre,
Math. Ann. 114 (1937), 305–315.
You may want to read about the early history of this theory here:
- Ch. Parsons. Developing Arithmetic in Set Theory without Infinity: some
historical remarks, History and Philosophy of Logic 8 (1987), 201–213.
That this is a reasonable setting for finitary mathematics has been argued extensively, see for example:
G. Kreisel. Ordinal logics and the characterization of informal notions of
proof, in Proceedings of the International Congress of Mathematicians. Edinburgh, 14–21 August 1958, J.A. Todd, ed., Cambridge
University Press (1960), 289–299.
G. Kreisel. Principles of proof and ordinals implicit in given concepts, in
Intuitionism and proof theory, A. Kino, J. Myhill, R. E. Veseley, eds.,
North-Holland (1970), 489–516.
That the theories are biinterpretable essentially means that they are the same, only expressed in different ways. More formally, we have a recursive procedure that assigns to each formula $\phi$ in the language of PA a formula $\phi^t$ in the language of ZFCfin and conversely, there is another recursive procedure that to each $\psi$ in the language of ZFCfin assigns a $\psi^{t'}$ in the language of PA. These procedures have the property that $\phi$ is provable in PA iff $\phi^t$ is provable in ZFCfin, and $\psi$ is provable in ZFCfin iff $\psi^{t'}$ is provable in PA, and moreover PA proves for each $\phi$ that it is equivalent to $(\phi^t)^{t'}$ and similarly ZFCfin proves of each $\psi$ that it is equivalent to $(\psi^{t'})^t$.
The translations are actually not too involved. In one direction, we can define in ZFCfin the class of natural numbers (the ordinals) and define its arithmetic operations the usual way, so that the resulting class satisfies PA. In the other direction, there is a nice argument: We can define in PA the relation $nEm$ iff there is a 1 in the $n$th place from the right in the binary representation of $m$. It turns out that the resulting structure consisting of the numbers seeing as the universe, with $E$ being membership is a model of ZFCfin. If we apply these procedures to the standard model $\omega$, we obtain $V_\omega$, and viceversa.
As for your question of whether one can do analysis in this setting, not quite. One can code some infinite sets in PA but not enough for all the arguments one needs in elementary analysis. A bit more formally, the subsystem of second order arithmetic known as $ACA_0$ proves the same arithmetic statements as PA, they are equiconsistent (provably in PA) and it is understood that it captures what we usually call predicativism. In particular, arguments requiring non-predicative definitions cannot be proved here. See:
Feferman, S. Systems of predicative analysis, i. The Journal of Symbolic Logic 29, 1
(1964), 1–30.
Feferman, S. Systems of predicative analysis, ii. The Journal of Symbolic Logic 33
(1968), 193–220.
The standard reference for systems of second order arithmetic is the very nice book:
You will find there many examples of theorems, results, and techniques in basic analysis that go beyond what $ACA_0$ can do. For example, the basic theory of Borel sets cannot be developed in $ACA_0$.
