Let's forget about $9-6\sqrt{2}$ for a second and just think about the expression your classmate thinks is useful:$$a^2-2ab+b^2.$$
And let's keep in mind our goal here. We're looking for something which is a perfect square (since we want it to play well inside a $\sqrt{\quad}$...
Well, this should remind us of $$a^2\color{red}{+}2ab+b^2=(a+b)^2.$$ But that "$-$" on the $2ab$ term is throwing me off! Is there any way to fix it?
This is where we get something for free from just doing a small change of variable: if we let $c=-b$, we get $$a^2-2ab+b^2=a^2+2ac+c^2.$$ That right hand side is of course just $(a+c)^2$, or better yet $(a-b)^2$. So we now know: $$\color{green}{\sqrt{a^2-2ab+b^2}=a-b}$$ (or rather, fine, $\vert a-b\vert$. FINE.).
That's why what your friend wants to do is reasonable. So nowthe question is: how do we do it?
Ultimately this can just feel like trial-and-error at first, but my instinct here is to say that "$-6\sqrt{2}$" looks a lot like "$-2ab$." Because they both have a minus sign. And $6$ is even. This doesn't work immediately, but when we factor out a $3$ things get much cleaner ...