I finished my PhD about a year ago and every now and then I get asked to referee a paper (perhaps once every two months from a reputable journal). I tend to accept as many as possible (except from crappy so-called "predatory" publishers, which I decline immediately) because I find refereeing papers is a good way to force myself to learn new things. There is also the moral obligation to referee a peer's paper to compensate for the time your peers spend refereeing yours.
My problem is that, at this early stage in my career, I'm familiar with the work I did during my PhD and little else. Since I try to be as fair as possible (I hate it when I get a report from someone who has clearly not understood the paper or just skimmed through the text), this means I spend a lot of time reviewing literature and trying to understand a paper before submitting a report. For something very close to what I have done in the past, I could finish the report in half a day. For something further away, I could be looking at anywhere from one to three days, depending on how familiar I am with the methods used and the length of the manuscript. As everyone in the academic world knows, it sometimes gets very busy (just "normal" busy otherwise), and these breaks tend to disrupt my "paid" job routine (I understand reviewing is also part of my unpaid duties).
I guess reviewing will get easier and quicker as I gain experience, but is it usual to spend so much time refereeing papers? What would be an acceptable compromise between a rigorous review and reconciling it with your paid duties?