I was learning about covariant and contravariant vectors due to special relativity, and it occured to me that we don't live in $\mathbb{R}^4$. I'll explain myself better. Consider the space of polynomials of degree $\leq n$. This is a vector space, and choosing ${1,x,x^2,...,x^n}$ as a basis makes perfect sense: those objects exist regardless of their coordinates. However, when we set up a coordinate system in classical mechanics (for example) we have a preesisting space (the "physical world"), in which we choose a point (which is going to be the $0$ of our identification with $\mathbb{R}^3$) and then a basis with which we are able to give coordinates to that point. So far so good, except that I have no idea what the "physical world" is without resorting to an $\mathbb{R}^3$ description in the first place.
Now, I thought of thinking about the "physical world" in a manifold sense: you have a topological space $(M, \tau)$ and you construct an atlas with one chart. In that sense, we can talk about coordinates and when we choose a reference point we are equipping the topological space of a manifold structure to talk unambiguously of points.
My question is: what's that topological space? What is the space I am equipping with a manifold structure?