+1 good question. This is a point of vagueness in a lot of heuristic descriptions of groups and symmetries. I think one confusing point is when you talk of rotation - do you mean it dynammicaly (here I'm rotating the square around, taking some time up to do so), or all at once (via the function that does the rotation).
The difference between the two is the difference between, one the one hand, a path in $SO_2(\mathbb{R})$ starting at the identity matrix and ending at a rotation matrix $M$, and, on the other hand, that rotation matrix $M$.
It's true that you can find paths of transformations in $SO(2)$ that bring the letter $R$ back to itself. However, these paths start and stop at the same point, namely the identity matrix.
When we talk about symmetries of subset $S \subset \mathbb{R}^2$, we generally interpret it the second way - as the subgroup $GL_2(\mathbb{R})$ (or the affine group if you are allowed translations) consisting of the matrices that fix the set $S$ and preserve all distances. (The choice between linear and affine symmetries also reminds me to tell you that you generally need to specify the symmetries of your object as belonging to some subgroup of the group of bijections, depending on the properties you want those symmetries to preserve. When people talk about symmetries in Euclidean space, they typically have in mind isometries.)
(Note that here there are two possible choices - do we want matrices $M$ so that as sets $MS = S$ or do we want to require that for all $s \in S$, $Ms = s$. When we talk about the symmetries of physical shape in Euclidean space we generally mean the former. The latter is much more restrictive. However, there are situations when we mean the latter (pointwise) option. For example, automorphisms of a field extension of $E$ over $F$ are taken to mean (among other requirements) that they fix $F$ pointwise.)
On the other hand, $SO(2)$ is a circle, and the path you describe (besides being set theoretically non-trivial), is also homotpically non-trivial, and indeed generates the fundamental group of $SO(2)$, which is $\mathbb{Z}$.