This is a question ($Q. 6$) from the book by Louis W. Shapiro, titled: Introduction to Abstract Algebra, from sec. $1.2$.
Let $S$ be a set, and let $G$ be the set of subsets of $S$.
If $A$ and $B$ are subsets of $S$,
then define $A * B = (A \cup B) - (A \cap B)$, that is, the subset of all elements in
$A$ or in $B$, but not in both.
(a) Show, using Venn diagrams if you wish, that $(A * B) * C = A * (B * C)$.
(b) Show that $(G,*)$ is a group.
(c) Show that $(G,*)$ is an abelian group.
(d) If $S$ has two or three elements, how many elements will be in $G$?
(e) If $S$ has $n$ elements, how many elements will be in $G$?
If $S$ has $n$ elements, then: (e) $|G|=2^n$. So, (d) is $4, 8$ respectively for set $S$ with $2,3$ elements.
(a) If $A * B = (A \cup B) - (A \cap B)$,
so, taking l.h.s. of $(A * B) * C = A * (B * C)$, get:
$(A * B) * C = ((A \cup B) - (A \cap B))*C$
$= ((A \cup B) - (A \cap B))\cup C - ((A \cup B) - (A \cap B))\cap C$
$= (A \cup B)\cup C - (A \cap B)\cup C - (A \cup B)\cap C + (A \cap B)\cap C$
$= A \cup B\cup C - (A \cap B)\cup C - (A \cup B)\cap C + (A \cap B)\cap C$
taking r.h.s. of $(A * B) * C = A * (B * C)$, get:
$A * (B * C) = A*(B \cup C) - A*(B \cap C))$
$= A\cup(B \cup C) - A\cap(B \cup C) - A\cup(B \cap C) + A\cap(B \cap C)$
$= A \cup B\cup C - (A \cap B)\cup C - (A \cup B)\cap C + (A \cap B)\cap C$
(b) Show that $(G,*)$ is a group, so need show that $(G,*)$ satisfies the four properties:
(i) Identity $(e)$ exists, s.t. for any set $A, A*e = e*A= A\cup e - A\cap e = A$.
As there is a set $\emptyset$ as subset of any set that satisfies this property, so $e= \emptyset$.
(ii) Inverse ($C$) exists for any element (set, $A$), s.t. $A*C = A\cup C - A\cap C = e\implies A\cup C = A\cap C\implies C = A$
(iii) Closure exists, as all sets are from the $2^n$ sets, and any union or intersection will still be in the superset.
Say, there are $3$ elements in $S$, so the subsets are $8$, labelled as :
x_1. $\lbrace 1,2,3\rbrace$,
x_2. $\lbrace 1,2\rbrace$,
x_3. $\lbrace 1,3\rbrace$,
x_4. $\lbrace 2,3\rbrace$,
x_5. $\lbrace 1\rbrace$,
x_6. $\lbrace 2\rbrace$,
x_7. $\lbrace 3\rbrace$,
x_8. $\emptyset$
$x_1 \cap x_2 = x_2, x_2\cap x_7 = x_8$, and so on.
But, request a more formal way to prove it.
(iv) Associativity property exists, as both commutativity (proved below in part (c)) and closure exist $\forall$ sets $A, B, C \in G \mid A*(B*C)= (A*B)*C$.
Request approach to prove (iv) by a better approach, as not possible to show failure by contradiction.
(c) To prove that $(G, *)$ is an abelian group means that $A*B$ yields the same results under swap ($B*A$).
$A*B = (A\cup B)-(A\cap B)$, while $B*A = (B\cup A)-(B\cap A)$
Due to the set union, set intersection & set difference being commutative operations; the result is the same under swap of sets for the operation.