When you say "normal exponentiation", you can think about natural numbers, real numbers, complex numbers, and so on.
Assuming that you mean natural numbers, we want a coherent behavior from infinite sets to the laws we have figured out on finite sets. In particular, we want $|A^B| = |A|^{|B|}$.
First let us review the basic laws of cardinal arithmetics:
$|A|+|B| = |\{0\}\times A\cup\{1\}\times B|$, that is addition of cardinals is the disjoint union of representatives.
$|A|\cdot|B| = |A\times B|$, that is multiplication is cardinality of the product.
$|A|^{|B|} = |A^B|$. This is the nugget of gold in this question, and we'll discuss this through the rest of this answer.
Of course we want these actions to behave in coherence with finitary actions. Multiplication is distributive over addition, exponentiation over multiplication, and so on and so forth. I will not get into that here.
Why do we want that? Because if we denote $n=\{0,\ldots,n-1\}$ for every $n\in\mathbb N$, then $k^n$ is just the amount of functions from any set of $n$ elements to any set of $k$ elements. This means that if $|A|=|C|$ and $|B|=|D|$ then we would like that $|A^B|=|C^D|$.
In other words, the cardinality is an equivalence relation and we want all the cardinal arithmetic to be independent of the choice of representatives (at least when finitely many sets are involved).
However, infinite sets - and in particular $\aleph$ numbers - behave much differently than finite sets. For example $|\{0,\ldots,n-1\}\cup\{n\}|>n$ while $|\mathbb N\cup\{-1\}|=|\mathbb N|$. Even more so, $|\mathbb N\times\mathbb N|=|\mathbb N|$. This means that some things, while working "the same" are going to work differently.
For your last question, we can notice that $\{f\colon\{1,2\}\to X\}$ would simply correspond to functions choosing two elements each time. This is just specifying ordered pairs in a very canonical way - $f(1)$ is the left coordinate and $f(2)$ is the right one.
This means that $A^{\{0,\ldots,n-1\}}$ can be thought of as the set of $n$-tuples from $A$, or $\underbrace{A\times\ldots\times A}_{n\text{ times}}$.
The laws of cardinal arithmetic tells us that $|A|\cdot|B| = |A\times B|$, so we have that $|A^{\{0,\ldots,n-1\}}|=|\underbrace{A\times\ldots\times A}_{n\text{ times}}|=\underbrace{|A|\cdot\ldots\cdot|A|}_{n\text{ times}}=|A|^n$. Of course, since we deal with infinite sets we are no longer guaranteed an actual increase of cardinality, but the laws work the same.
What if $|A|$ and $|B|$ are infinite? Well, we already have the laws for how we wanted exponentiation to behave, now we just need to apply them. We can think of $A^B$ as $B$-tuples of $A$, that is every element of $B$ is an index of a sequence from $A$. These sequences need not be finite, nor countable - not even well ordered.
And indeed, we see that if $A=\{0,1\}$ then the rule $|P(B)| = |\{0,1\}^B|=|\{0,1\}|^{|B|}=2^{|B|}$ holds as we wanted it to hold.