$A \subset \ell^2$ is closed if and only if whenever $x^{(n)}$ is a sequence of elements of $A$ and $x^{(n)} \to x$ in the $\ell^2$-norm we have $x \in A$.
Now, if $x^{(n)} \to x$ in $\ell^2$ then we have that for each $k \geq 1$, $x_k^{(n)} \to x_k$ in $\mathbb{R}$ (i.e. convergence in $\ell^2$ implies convergence of the individual coordinates). This makes it easy to see that your example of a closed subset is indeed closed. If $x^{(n)} \to x$ in $\ell^2$ then $x_k = \lim_{n \to \infty} x_k^{(n)} = 0$ for $k \geq 4$ since $x_k^{(n)} = 0$ for all $n \geq 1$ and $k \geq 4$.
The standard example of a subspace of $\ell^2$ which isn't closed is $$c_{00} = \{ x \in \ell^2 : x_k = 0 \text{ for all but finitely many } k\}.$$ It's easy to check that this is a subspace and in fact it is dense in $\ell^2$ since if $x = (x_1, x_2, x_3, \dots) \in \ell^2$ then $x^{(n)} = (x_1, \dots, x_n, 0, 0, 0, \dots) \in c_{00}$ converges to $x$ in $\ell^2$ as $n \to \infty$. So $c_{00}$ is dense in $\ell^2$ and is not all of $\ell^2$ so cannot be closed.