First of all and most importantly: Mathematical existence has nothing whatsoever to do with physical existence. Just because something does not "exist physically" does not mean it cannot exist mathematically. Mathematics was partly created to model physics, but this does not mean that mathematics cannot make simplifications (like believing there are points). But even then, applications of mathematics go way beyond "just" physics. You can also think of it this way: Mathematics is so amazing, besides the physics you are used to it may explain any kind of physics you can come up with in your head.
A mathematical point (in let's say two dimensions) is just a pair $(x_0,y_0)$ consisting of real numbers $x_0,y_0$. If you have another point $(x_1,y_1)$, you get a line through these two points. It's the set of all points $(x,y)$ with $y=ax+b$, where $a = \frac{y_0-y_1}{x_0-x_1}$ and $b = y_0 - mx_0$. Of course, there are infinitely many points $(x,y)$ on said line (take any $x$ and calculate the corresponding $y$). But even between $(x_0,y_0)$ and $(x_1,y_1)$, there are infinitely many points. Take any $\lambda\in [0,1]$, then $(\lambda_0 + (1-\lambda)x_1, \lambda y_0 + (1-\lambda) y_1)$ will be a point "between" $(x_0,y_0)$ and $(x_1,y_1)$.
The concept of a point can be generalized to abritary dimensions ($\mathbb{R}^n)$, other rings ($\mathbb{Z}^n$, $\mathbb{Q}^n$, $\mathbb{C}^n$, ...), later to points in a metric space and "finally" to points in a topological space. Then points need not actually represent anything physical. For example you could take the vertices of a graph (that is a thing with vertices and edges between those vertices) to be your points and measure distance between these vertices (how many steps do I need to get from one vertix to another vertix?). This yields a metric space, which does not have anything directly to do with physics, but which is still very interesting and important.
There are also other approaches to formalizing the concept of 'point', for example incidence geometry: It abstractly deals with points and lines, and may be used to formalize different kinds of geometries synthetically.