Your job, as an academic, is twofold:
- Do amazing research.
- Write it up and convey your results to other people.
It sounds like you view (2) to apply only to others who are already in your field and are completely familiar with everything you've done. You will find that, in the entirety of your field, there are probably just a few dozen people who are intimately familiar with every details of your field, and only a few hundred who are really familiar with what you do. Everyone else — academic, layperson, village idiot — will require an explanation, and you should do them the favor of explaining it to them.
To more directly answer your question, you should always have two answers ready to the question "what do you do?":
- An elevator pitch, as described elsewhere. This should take ~15-30 seconds to say and would give a very high-level overview of your work.
- A more in-depth explanation, which would take about 3-5 minutes, which conveys what you do in more detail. Generally speaking, less time than that and you can't convey any useful information, more time than that and you're giving too many details. After your in-depth explanation, either the person will say, "oh", and move on, or they'll ask questions and you can have an intellectual discussion.
(Note: the following point is somewhat debatable.) I've found for myself that it helps to visualize the person you're talking to as paying your salary; if you're on a publicly-funded research grant (i.e., any governmental grant), their taxes are funding your research. It gives some perspective.