I wholeheartedly agree with every word quantumzorn has said. MIT's Opencourseware classes have proven to be an incredible help in learning mathematics to me as well ( I apply it all to Physics accordingly.) I've made the same mistake he has, but another approach that is available--and I won't recommend it. It's just the approach I take--is to start out with what you're wanting to learn, and look at an introductory text for what you're wanting to learn. Pretty quickly you'll find out whether or not you have the mathematical backing to do it or not. Nevertheless, once you get to a point where you have no clue what's going on, figure out what branch of mathematics is being applied there, and look at an introductory text for that. From there, it's just rinse and repeat until you get to a point where you can build upon knowledge you have, or gain new knowledge.
This method, however, can be very expensive and time consuming. For example:
I've been wanting to learn the mathematical foundations of Einstein's Relativity. This is arguably higher level mathematics ( Mostly Tensor Calculus ). Tensor Calculus was my top level math branch. Coming down from there, I learned that tensor Calculus is derived from Differential Geometry. My Second level down from Relativity. From there, I figured out that Differential Geometry is laden with Vector Analysis ( Vectors are merely a type of tensor. Level 3. Fortunately, I was already familiar with Vector Analysis, but vector Analysis picks up at the end of most college calculus courses ( Calc III ), which happens to be multivariable calculus. Level 4. Which then at the point, it's just a step down to single variable calculus, algebra, then your high school level mathematics.
One of the benefits of this approach though is that you'll quickly be able to figure what branch of mathematics something is, and sub branches required to learn what you want rather quickly. The way I've described my Relativity experience appears to be just a linear progression of doctrines of math. Most of the time though it isn't that straight forward though. I find it very helpful to make tree diagrams--probably pretty obvious that I was alluding to that. I also came to find that tensor not only include vectors, but matrices. What is the mathematics of matrices in general? Linear Algebra. That would go on level 2 alongside Differential Geometry. I'd go so far as to say that those two course can be taken concurrently, but I digress.
But once you've learned enough to ( in my case ) get past a certain part of a Relativity text, continue on until you hit a point in which something doesn't make sense, and then rinse and repeat. This process, again, can be very time consuming, but the more often you do it, the quicker you'll get at it. I'm only 19, and started with just an introductory level calculus text with just a high school knowledge of mathematics--and I wasn't even good at math to start lol--when I was 18, and utilizing MIT's resources, as well as any book I could get my hands on, I have no problem with figuring out what I need to learn for what I want to do. I won't say when doing this I was doing this religiously, it was more of--just like your case--kind of a hobby, and even now that I don't have much time, this method is still very practical.
Good luck!