In set theory $\in$, the membership relation, is the basic building block of everything else. We write $a\in A$ to indicate that $a$ is an element of the set $A$. If $A$ is small enough we can write $A=\{x,y,z\}$ and then $a\in A$ says that $a=x$ or $a=y$ or $a=z$. However often sets are too big (or too general) for us to write in such way (e.g. $\mathbb N$ is an infinite set).
Elements could be numbers, but they could also be sets. In fact in modern set theory everything is a set, and we can represent the natural numbers, real numbers, or so on as sets.
The collection defined $A=\{x\mid x\notin x\}$, is the collection of all objects which are not members of themselves. However since we know that only sets have members this means that this is the collection of all sets which are not members of themselves.
It can be shown that $A$ is not a set. The way to do that is to argue, if $A$ was a set, then either $A\in A$ and then $A$ is a member of itself and by the defining formula of $A$ we have that $A\notin A$ - which is a contradiction; or $A\notin A$ and then by the defining property of $A$ we have $A\in A$ and again we derive a contradiction.
The only conclusion is that $A$ is a collection which is too big to carry the title set. This is known as the Russell paradox of naive set theory, and was one of the reason axiomatic approaches were taken to describe what is a set.
$symbols between letters, and it is possible (and preferable) to use\{\}for{}rather than leaving them outside the LaTeX part. – Asaf Karagila Nov 10 '12 at 09:05