Your question is based on a wrong assumption, namely that fast reviews mean reject. I myself have accepted reviews and finished the review a day later with an accept recommendation. This was because I happened to have time that day. As a reviewer, you cannot spend an inordinate amount of time on a review. In Computer Science, if I am familiar with the subject, I can usually finish a review in a few hours. I usually wait for a night and a final second look before I submit a review. If I am not that familiar with the area, then of course it takes longer. I am not holding myself up as a good example, but just to assure you that a positive review might be done within five days based on my own experience.
If your field is Mathematics and you write in the typical terse style, then reviews are just bound to take longer, because reading Mathematics papers involves much more work. But then, when I was still a working Mathematician, the (bitter and cynical) joke around my colleagues was that the average number of readers of a Mathematics paper was less than 1.0, including reviewers and editors. Even in this case, if the review request arrives to the reviewer who has time (or wants to procrastinate grading, ...) the reviewer might do all the work within a few days.
Finally, you also assume that a single negative review will sink your paper. Even at a very prestigious journal, where one negative review will usually lead to a reject, the editor reads the reviews, reads the self-assessment and comments to the editor (such as, "I think this is non-sense but I have not done Beginning of Life Biology for decades and am now working in Experimental Archeology of South New Zealand, maybe you confused me with my cousin"), and only after looking at all the input from the reviewers will the editor make a decision.