Contrary to belief, a rigorous calculus text is right up your alley! Books such as Apostol I,II and Spivak's calculus start of by developing your understanding of the real numbers which will require you to work on your algebra skills. To understand physics outside of the university level (a first year's course) you'll need much more than calculus.
A really good understanding of multivariable calculus will get you through most undergraduate physics courses. Therefore a book which takes you through the whole calculus sequence is more desirable since you don't want to waste time trying to adjust to a new writer's style + notation.
Any of these two books mentioned above provides a nice 2 for 1 in your case since they'll make you improve your algebra skills before the calculus starts.
Personally, I didn't learn from the books I've suggested, but I wish I would have. I've noticed that although I know how the ideas in calculus work (since I do a lot of differential geometry, etc), I am sometimes fuzzy on the analysis to rigorously prove arguments that I know to be true. As mathematicians, we care very much about details, especially when the big picture is clear and we want to smoothen out an argument. However, a physicists may not be to concerened about these things, so I also suggest reading other posts related to this one, as other users have suggested.
$\textbf{Comment}$:I believe I am okay in saying this, check abebooks for copies of the texts I mentioned above. They sell international copies which are much cheaper that those you'll see on amazon. I hope this helped.