My first try was to set the whole expression equal to $a$ and square both sides. $$\sqrt{6-\sqrt{20}}=a \Longleftrightarrow a^2=6-\sqrt{20}=6-\sqrt{4\cdot5}=6-2\sqrt{5}.$$
Multiplying by conjugate I get $$a^2=\frac{(6-2\sqrt{5})(6+2\sqrt{5})}{6+2\sqrt{5}}=\frac{16}{2+\sqrt{5}}.$$
But I still end up with an ugly radical expression.
Here's a useful formula for this kind of problems:
$$\sqrt{a \pm \sqrt{b}} = \sqrt{\frac{a + \sqrt{a^2 - b}}{2}} \pm \sqrt{\frac{a - \sqrt{a^2 - b}}{2}}$$
where we have $a,b \ge 0$ and $a^2 > b$
$$a=\sqrt{6-\sqrt{20}}$$
$$\implies a^2=6-\sqrt{20}$$
$$\implies(a^2-6)^2=20$$
$$\implies a^4-12a^2+16=0$$
$$\implies (a^2-2a-4)(a^2+2a-4)=0$$
so it will be one of the four possibilities $\pm\sqrt{5}\pm1$, and since $\sqrt{6-\sqrt{25}} \lt \sqrt{6-\sqrt{20}} \lt \sqrt{6-\sqrt{4}}$, you want the one which is in $[1,2]$
Let $\sqrt{6 - \sqrt{20}} = \sqrt{a} - \sqrt{b}$, where $a$ and $b$ are rational numbers. Squaring both sides of the equation $$\sqrt{a} - \sqrt{b} = \sqrt{6 - \sqrt{20}}$$ yields \begin{align*} a - 2\sqrt{ab} + b & = 6 - \sqrt{20}\\ a - 2\sqrt{ab} + b & = 6 - 2\sqrt{5} \end{align*} Matching rational and irrational parts yields the system of equations \begin{align*} a + b & = 6 \tag{1}\\ -2\sqrt{ab} & = -2\sqrt{5} \tag{2} \end{align*} Solving equation 2 for $b$ yields \begin{align*} -2\sqrt{ab} & = -2\sqrt{5}\\ \sqrt{ab} & = \sqrt{5}\\ ab & = 5\\ b & = \frac{5}{a} \end{align*} Substituting $5/a$ for $b$ in equation 1 yields \begin{align*} a + \frac{5}{a} & = 6\\ a^2 + 5 & = 6a\\ a^2 - 6a + 5 & = 0\\ (a - 1)(a - 5) & = 0 \end{align*} Hence, $a = 1$ or $a = 5$.
If $a = 1$, then $b = 6 - a = 5$, in which case $$\sqrt{6 - \sqrt{20}} = \sqrt{1} - \sqrt{5} < 0$$ which is impossible since the principal square root of a positive number must be positive.
Thus, $a = 5$ and $b = 6 - a = 1$, so $$\sqrt{6 - \sqrt{20}} = \sqrt{5} - \sqrt{1} = \sqrt{5} - 1$$
Check: Observe that $\sqrt{5} - 1 > 0$. Moreover, $$(\sqrt{5} - 1)^2 = 5 - 2\sqrt{5} + 1 = 6 - 2\sqrt{5} = 6 - \sqrt{20}$$ Hence, $$\sqrt{6 - \sqrt{20}} = \sqrt{5} - 1$$ as claimed.
There's actually a general formula for these kinds of expressions. Namely$$\sqrt{X\pm Y}=\sqrt{\frac {X+\sqrt{X^2-Y^2}}2}\pm\sqrt{\frac {X-\sqrt{X^2-Y^2}}2}$$ Where $X,Y$ are real numbers. Simply substituting $X=6$ and $Y=\sqrt{20}$ gives the proper denesting. The proof of this is quite simple. Assume that$$X\pm Y=\left(\sqrt A\pm \sqrt B\right)^2$$and expand via binomial theorem. Collecting terms, you will end up with two equations from which you can solve for $A$ and $B$ in terms of $X$ and $Y$.
In this particular problem, you can pretty much guess the answer.
$$\sqrt{6-\sqrt{20}}=\sqrt{6-2\sqrt{5}}$$
Now, suppose that the $-2\sqrt{5}$ was the middle term of a perfect square trinomial, where $x = \sqrt{5}$. In other words, that middle term is $-2x$.
What would the first and last term look like? Obviously it would be $x^2$ and $1$ respectively.
$x^2 - 2x +1 = (x-1)^2$
Substituting $\sqrt{5}$ for $x$ we have... $$x^2 - 2x +1 = (x-1)^2$$ $$\sqrt{5}^2 - 2\sqrt{5} +1 = (\sqrt{5}-1)^2$$ $$5 - 2\sqrt{5} +1 = (\sqrt{5}-1)^2$$ $$6 - 2\sqrt{5} = (\sqrt{5}-1)^2$$ and taking the square root of both sides we have $$\sqrt{6-2\sqrt{5}} = \sqrt{5}-1$$
It's almost like somebody just make up that problem to work out cleanly like that. ;)
