$\mathtt{ "Slot\; Method"\; Answer}$
I see that you attempted to use what I call the "slot method" which, as you can see here can be a huge shortcut. However, unless you are careful, in more complex situations, you can miss nuances.
I have already given a conventional answer, so here I shall be brief, following your answer with the necessary refinements.
P($3$ sets remain) $= 1 * \frac{18}{20}\frac{15}{19}\frac{12}{18} = \frac9{19}$
$\mathtt{(Simple\; Version\; of\; Slot\; Method)}$
P($4$ sets remain) $= \binom42 *1*\frac2{20}\frac{18}{19}\frac{15}{18} = \frac9{19}$
$\mathtt{(We\; first\; need\; to\; choose\; which\; two\; will\; be\; together)}$
$\mathtt{Five\; sets\; remaining\; can\; discard\; 2-2\; or\; 3-1}$
P($5$ sets remain) $= 3*1*\frac {2}{20}\frac{18}{19}\frac{2}{18} +4*1*\frac{2}{20}\frac{1}{19}\frac{18}{18} = \frac{1}{19}$
ADDED: Explanation of multiplication factors
You can consider the blocks as forming unlabeled teams (except where they become labeled by different sizes), thus
- $2-1-1:\; \binom42\binom21\binom11\div 2! =6$
- $2-2:\; \binom42\binom22\div2! =6$
- $3-1:\; \binom43\binom11 = 4\;$ (No divisor needed as labeled by size)
- By the same process, you should understand why the $1-1-1-1$ configuration doesn't need a multiplication factor (= has a m.f. of $1$), as $\binom41\binom31\binom21\binom11\div4! = 1$
I have explained the multiplication factors at length, but they are actually quite simple, (break into teams the normal way and divide by $r!$ where $r$ unlabeled teams are there)