There is no "the 4th dimension" in mathematics. Mathematically speaking, it makes perfect sense to speak of any number of spatial dimensions, none of which need to have anything to do with time. (And in fact some physical theories require as many as 25 spatial dimensions!)
At the simplest level, to talk about four spatial dimensions we simply use four coordinates $(x, y, z, w)$ instead of three coordinates $(x, y, z)$. The ordinary space we see around us also has a notion of distance given by the Pythagorean theorem
$$\sqrt{(x_1 - x_2)^2 + (y_1 - y_2)^2 + (z_1 - z_2)^2}$$
and similarly in four dimensions we can talk about the distance
$$\sqrt{(x_1 - x_2)^2 + (y_1 - y_2)^2 + (z_1 - z_2)^2 + (w_1 - w_2)^2}.$$
In mathematics this is just another example of a metric space, or more specifically a Riemannian manifold.
When physicists use four coordinates $(x, y, z, t)$ and they want the fourth coordinate to represent time, and also they are not ignoring special relativity, then rather than the ordinary distance the relevant number is
$$\sqrt{(x_1 - x_2)^2 + (y_1 - y_2)^2 + (z_1 - z_2)^2 - (c t_1 - c t_2)^2}$$
where $c$ is the speed of light (note the difference in sign!). This is the proper distance between two spacetime events, and it behaves very differently from ordinary distance (e.g. it may be zero for some pairs of events, and it is not well-defined for some pairs of events). Mathematically we have a Lorentzian manifold.