There are many set theories, but the most common one, is ZFC. I will
talk only about ZFC, because I do not know every set theory that exists
and I do not want to make generalizations that do not hold. ZFC is
a theory in first order logic. That means that it has not only the
symbol "$\in$", but also all symbols that the language of first
order predicate calculus has, like the symbols of variables, the parenthesis
"$($" and "$)$", the logical connectives "$\neg$", "$\land$", "$\vee$", "$\rightarrow$", "$\leftrightarrow$", or some of them, the quantifiers "$\forall$" and "$\exists$", or at least one of them and perhaps, the equality symbol "$=$".
All the symbols that belong to the language of logic, are called "logical"
symbols.
Of course, one cannot express everything using logical symbols alone.
In order to apply logic in other fields of mathematics, one must expand
its language with all the symbols that are necessary for that field.
Those are called "non logical" symbols and they are listed in
the "signature" of the language. Because the logical symbols are
taken for granted, it is convenient to describe a language only by
its non logical symbols. When somebody says "the language of set
theory has only the symbol "$\in$" ", he means that it has
only "$\in$" in its signature. That is: beyond the logical symbols,
the only symbol is "$\in$".
Now, why does ZFC only needs "$\in$" as non logical symbol? Because
it is enough to formulate all the axioms of that theory, and because
everything else results from the axioms. So, every statement of set
theory can be expressed by the logical symbols and "$\in$", alone.
Then, what about symbols like "$\varnothing$", "$\cup$",
"$\subseteq$" etc? Well, those symbols are used for convenience,
but they are not necessary. Everything that can be written with them,
it can also be written without them. For example, "$a\subseteq b$"
is equivalent to: "$\forall x\left(x\in a\rightarrow x\in b\right)$".
The first is just an abbreviation of the second. Or, lets take $a\cup b\neq\varnothing$.
This is equivalent to $\exists c\left[\forall x\left(x\in a\vee x\in b\leftrightarrow x\in c\right)\wedge\exists y\left(y\in c\right)\right]$.
It is not forbidden to include non necessary symbols to a signature - in fact, we often do - but it is not necessary either. If we do not include such symbols, the formal language does not have them, and every formula that is made of them, does not belong - strictly speaking - to the formal language. In practice, we often use them, but it is easy to translate our formulas in the strictly formal form, whenever is needed.
In your question, you also mention natural numbers. There are is a
definition of natural numbers, from the polymath John von Neumann,
based on ZFC. He defines 0 as the empty set, and the successor of
a number $n$ is $n\cup\{n\}$. This gives: $0=\varnothing$, $1=\{0\}=\{\varnothing\}$,
$2=\{0,1\}=\{\varnothing,\{\varnothing\}\}$, $3=\{0,1,2\}=\{\varnothing,\{\varnothing\},\{\varnothing,\{\varnothing\}\}\}$...
Then, the set $\mathbb{N}$ of all natural numbers, is the smallest
set that contains $\varnothing$, which is closed under the function
$s(n)=n\cup\{n\}$. In this way, every proposition that contains numerals,
can be translated to a proposition that contains only logical symbols
and "$\in$".
However, if you do not use von Neuman definition, and your numerals
have a different meaning, which is not definable inside ZFC, then
you may have to include some of them in the signature, together with
other necessary symbols, from which you will define the rest. In that
case, you have created an extension of your language, which is not
the language of set theory any more.
Is my formulation correct?
– Delta Psi May 14 '21 at 09:57