This is a common error.
In a group $G$, with $a,b\in G$, the commutator of $a$ and $b$ is $[a,b]=aba^{-1}b^{-1}$ (note: it can also be defined as $[a,b]=a^{-1}b^{-1}ab$; some formulas change a bit, but the basic properties are the same; the difference is that with your definition, $[a,b]$ is the unique element of $G$ such that $ab=[a,b]ba$, whereas with the other definition it is the unique element such that $ab=ba[a,b]$).
However, the commutator subgroup $[G,G]$ of $G$ is not the set of all commutators. It is the subgroup generated by all commutators. That is,
$$[G,G] = \Bigl\langle [a,b]\,\Bigm|\,a,b\in G\Bigr\rangle,$$
and not $\{[a,b]\mid a,b\in G\}$.
In fact, it is false that the set of commutators is a subgroup in general, though specific examples may be difficult to find, as the smallest group for which the set of commutators is not a subgroup is of order 96; this was established in Robert Guralnick's dissertation. In fact, there are two nonisomorphic groups of order 96 in which the set of commutators is not closed under products.
There is a nice note of Marty Isaacs in the Monthly providing an infinite family of examples of finite groups in which the commutator subgroup contains elements that are not commutators: you can find it here courtesy of Pete L. Clark. It is
- Commutators and the commutator subgroup by I.M. Isaacs, American Mathematical Monthly 84 (no. 9) (Nov 1977), pp. 720-722
More information in this math.overflow question and this question on this very site