$\text{SU}(2)$ is in fact diffeomorphic to the $3$-sphere $S^3$, rather than the $2$-sphere (which has the wrong dimension; recall that $\mathfrak{su}(2)$ is $3$-dimensional); this comes from its identification with the unit quaternions. $\text{SO}(3)$ is diffeomorphic to the real projective space $\mathbb{RP}^3$, which is the quotient of $S^3$ obtained by identifying antipodes.
In general, $\text{U}(n)$ can't be identified as a more familiar-looking manifold. It is an "iterated extension" of the odd-dimensional spheres $S^1, S^3, ..., S^{2n-1}$, and in fact is rationally homotopy equivalent to the product $S^1 \times S^3 \times ... \times S^{2n-1}$. This means in particular that it has the same rational cohomology and rational homotopy groups as this product; however, it is in general not not homeomorphic or diffeomorphic to this product. "Iterated extension" means that the unitary groups fit into fiber sequences which are ultimately built from odd spheres, starting with
$$\text{SU}(n) \to \text{U}(n) \xrightarrow{\det} S^1$$
and continuing with
$$\text{SU}(n-1) \to \text{SU}(n) \to S^{2n-1}.$$
The first sequence is even a short exact sequence of Lie groups, and it splits smoothly, so $\text{U}(n)$ is diffeomorphic to $\text{SU}(n) \times S^1$; in particular $\text{U}(2)$ is diffeomorphic to $S^3 \times S^1$. But this is not an isomorphism of groups ($\text{U}(n)$ is a semidirect product rather than a direct product), and the corresponding statement for the other fiber sequences should be false, although I haven't verified this. I do know that $\text{SU}(3)$ is not diffeomorphic, and in fact is not even homotopy equivalent, to $S^3 \times S^5$; see this MO question.
The infinite-dimensional unitary group is not a manifold in the usual sense, although it is a Hilbert manifold. Kuiper's theorem implies that it is weakly contractible, so from the perspective of homotopy theory it looks like a point. But of course what actually appears in physics is not the unitary group but the projective unitary group; the infinite-dimensional projective unitary group is an Eilenberg-MacLane space $K(\mathbb{Z}, 2)$. This comes from its identification as a quotient of a weakly contractible space, namely the infinite-dimensional unitary group, by a free action of $\text{U}(1)$; hence it is a model for the classifying space $B \text{U}(1)$, and $\text{U}(1) \cong S^1$ itself is a $K(\mathbb{Z}, 1)$.
A closely related space called the stable unitary group is also very interesting from the perspective of homotopy theory; its homotopy groups are $2$-periodic, which is one way of stating complex Bott periodicity.