In general I will agree with the other answers that there is little correlation between paper quality and review time, once you have made it past editorial rejection. At that point, the dominant factor in time is the arbitrariness of peer reviewer responsiveness and editorial distraction, since reviewing papers is a lower-priority task for most active researchers.
In my experience as an editor, however, there is one class of manuscript that does tend to take unusually long to get responses on: not-very-good presentations of potentially interesting work.
If the work was presented well, it would be easy to provide a review based on quality. If the presentation very bad, then it would be easy to reject on the basis of presentation. In between, however, is an uncanny valley filled with work that is slow and painful to review.
Unless you are getting complaints about confusing presentation from your reviewers, however, you shouldn't guess that your work is in the uncanny valley. Instead, you should assume that you're just experiencing the arbitrariness of reviewing time.