Mathematics is difficult and if you lack confidence it'll be hard to keep going. Of course, not many people have confidence that they know a lot - actually most of us know that we know very little in the scope of things. However, I've seen that good mathematicians seem to be confident in their abilities to learn given enough effort and patience. Some learn fiendishly quickly and it can be discouraging to be among them but others like me really need our time to sit down and explore the definitions, theorems, etc. To survive in this game you need to be confident in yourself and a great thing to be confident about is work ethic. If you are able to put in a couple of hours into solid learning every day, you'll tackle almost any subject.
About studying itself - there are lots of different ways to go about it and some are more mind-numbing than the rest. The texts are dense, especially Rudin. You can't understand them the way you understand a novel when you read it. Actually, if the text is new to you, you cannot even read it with any appreciable speed. A good way to learn is to take it very slowly, sentence by sentence. Rip each sentence apart and see that you know each word and expression, and that you get the meaning of the words combined. Make diagrams and check your intuition. It takes keeping your foot on the brake because it's really easy to just run-off onwards and glaze over. It might seem like it would take forever to learn this way but it's more resilient.
You lost motivation in your studying because you faced many big failures in a short amount of time. Suddenly a lot of long hours of studying, writing applications and proof-reading letters seem like they have been for nothing. All of a sudden, you feel yourself burned out. I see nothing wrong in taking the kind of break that you took because it can be traumatizing. If you come back to math and feel still deep anxiety, I know how you feel. The problem is that math is difficult and we face a ratio of many failures for each success, and this seems to be mimicked in my friends' career paths too. This kind of ratio is absolutely toxic to confidence. You need to redefine what success means to you.
In my undergrad my successes were usually finishing an assignment, doing great on an exam, and occasionally solving a hard homework problem that bothered me for days. But as the years progress I faced more and more difficult problems. All of a sudden, assignments were incomplete, problems had holes in them, exams were not stellar, and hard problems remained unsolved. I lost a lot of confidence and was in a situation similar to yours several times. It's something a lot of math students face (at least among my peers). They all seem to meet their match that makes them or breaks them. What it takes is a shift of the mind. You are a mathematician and you can make your own definition and properties.
So here it goes. When there's failure in everything you do you gotta look for the small successes. You understood a paragraph - that's a success. You understood the idea behind a proof - that's also a success. You find an interesting property of some function - another win. One case down out of five - another success. Basically, when the stairs become too tall to scale, you gotta make your own ladder so you can climb up more easily.
When you focus on the small successes, there will be lot of them and failure will not bother you as much anymore. All of a sudden it wouldn't matter so much if the whole assignment is in or if you understand a whole chapter. You'll enjoy making the little steps and they'll eventually take you much further than trying to make bigger leaps.
I hope I'm not reading too much into your words. When I read your question I kind of saw myself as I was a few years ago and I wish someone would have told me what I know now.
