You are right that Cayley's theorem applies to every group, assuming we talk about bijections. But what are permutations? Are they different from bijections? And why we mostly consider them on finite sets?
This is of course a matter of nomenclature. However typically when we think about permutations, we consider bijections over a finite set. Because when we think about permutations, we also think about their properties, for example every permutation has a sign. It is an important property, but it does not generalize to arbitrary bijections. If we want to keep good properties of permutations, we can define it as follows: a permutation is a bijection $f:X\to X$ such that $f(x)=x$ for all but finitely many $x\in X$. Such permutations have well defined sign, and we can say more about a group of permutations $S(X)$. For example it has a subgroup $A(X)$ of index $2$, which is simple. The more general group of all bijections $Bi(X)$ is waaay too complicated. The problem is that Cayley's theorem does not hold for $S(G)$, except for the special case when $G$ is finite.
And so, in infinite case it is a trade-off: either we have "good" bijections, but then Cayley doesn't hold or Cayley holds, but we have "bad" bijections, too wild for proper analysis.