P.S Why cannot not we just cancel ab out of the middle of these expressions? Why can we only cancel "on the left" and "on the "right"? Could somebody explain that to me (if it is possible, could you refer to definitions/ theorems when doing that)?
We can multiply the expressions by $a^{-1}$ and $b^{-1}$ , as we have a group and it is one of the properties of a group? Am I right?
When we have $xyz=x'yz'$ and cancel out the middle factor, we can only do that if we know that the commutes with $x$ and $x'$, or with $z$ and $z'$. From being so accustomed to working with commutativity, we lose sight of this fact. In reality, we can only cancel from the left or right, and the only way we can do more is if we can move a factor to the left or right.
As for your problem, $${(ab)^5\over (ab)^3}={a^5b^5\over a^3b^3}=a^2b^2.$$
But, $${(ab)^5\over(ab)^3}=abab.$$
Hence, $$abab=(ab)^2.$$ Now, multiply on the left by $a^{-1}$ and on the right by $b^{-1}$.
The middle cancellation property does not hold for every group. See this example.
In fact a group possesses the middle cancellation property if and only if it is abelian:
$(\Rightarrow)$ If the middle cancellation property holds, then for any $a$, $b$ we have $aa^{-1}b=b=ba^{-1}a$, so $ab=ba$.
$(\Leftarrow)$ If the group is abelian and $axb=cxd$ then $$ab=x^{-1}xab=x^{-1}axb=x^{-1}cxd=x^{-1}xcd=cd . $$
