Given that $G = \langle x,y | x^5=y^4=1,yx=x^2y\rangle$, how would I prove $G$ is a non-abelian group of order $20$ (and not isomorphic to $D_{10}$)?
Here's what I have so far:
$y^4=1$ so $xy = y^4xy = y^3(yx)y = y^3x^2y^2$
Honestly I've tried some more adding onto the right and left side, but I keep getting stuck. I'm assuming the best way to go forward is to try and prove $yx \neq xy$? Could someone push me in the right direction?
Consider the group $G' = \mathbb{Z}/5\mathbb{Z} \rtimes \mathbb{Z}/4\mathbb{Z}$, where the group multiplication is defined as $$ (\overline{a},\overline{b}) \star (\overline{a'},\overline{b'}) = (\overline{a+2^ba'},\overline{b+b'})\, , $$
(which is a valid multiplication because $2^4 \equiv 1 \mod 5$). If we set $x = (\overline{1},\overline{0})$ and $y = (\overline{0},\overline{1})$, then the relations of the presentation are satisfied, so there exists a surjective map from $G$ to $G'$. However, it is not so difficult to see that $|G| \leq 20 = |G'|$, so in fact we have $G \simeq G'$.
I leave the fact that $G$ is nonabelian and not isomorphic to $D_{10}$ as an exercise.
I'll use $C_n$ as notation for the cyclic group of order $n$.
I claim that $G$ is isomorphic to $C_4 \rtimes C_5$, where the generator of $C_4$ acts on $C_5$ via squaring, hence it is non-abelian of order $20$.
Indeed, this is immediate from the fact that $C_4 = \langle y \mid y^4=1 \rangle$ and $C_5 = \langle x \mid x^5=1 \rangle$ and how a semidirect product looks like in terms of generators and relations, compare Remark 1.7.1 in these notes.
Here's a direction you can start with: first of all, note that while the relation $yx = x^2y$ doesn't say that $x$ and $y$ commute, it still lets you move any $x$s "to the left of" all $y$s. In particular, this means that every element of the group can be expressed in the form $x^iy^j$ — and since we have $x^5=1$, $y^4=1$ then we can even impose the restrictions $0\leq i\leq 4$, $0\leq j\leq 3$. Now, this shows you that the group has at most 20 elements; for convenience I'll express an element $x^iy^j$ as $\langle i,j\rangle$. Next, you can use the generator relations to express the group product of $\langle i,j\rangle$ and $\langle k,l\rangle$ with 'arithmetic operations' on $i,j,k,l$; this should also let you easily compute the inverse of any $\langle i,j\rangle$ in that form. It shouldn't be too hard to show that all of these elements are distinct; to show that your group isn't $D_{10}$, you can consider the number of elements of order $2$ in each.
