As noted, the result is false in the infinite setting, unless $X$ is closed under inverses, or you know that every element of $X$ is torsion; or $N$ is itself finite. The problem is that in general you cannot express $x^{-1}$, with $x\in X$, as a product of elements of $X$ unless $x$ has finite order.
For an explicit counterexample, start with the group of invertible matrices with coefficients in $\mathbb{Q}$ of the form
$$\left(\begin{array}{cc}
a&b\\0&c\end{array}\right),\qquad ac\neq 0.$$
Let $N$ be the subgroup of matrices with $a=c=1$, $b\in\mathbb{Z}$; and let
$$y = \left(\begin{array}{cc}
\frac{1}{2}&0\\0&1\end{array}\right).$$
Finally, let $G=\langle N,y\rangle$, and set $X=\{y\}\cup\{z\mid z\in N\}$.
Since $N$ is a subgroup and
$$y^{-1}\left(\begin{array}{cc}
1&b\\
0&1\end{array}\right)y = \left(\begin{array}{cc}
1&2b\\
0&1\end{array}\right),$$
it follows that $x^{-1}Nx\subseteq N$ for every $x\in X$.
However,
$$y\left(\begin{array}{cc}
1&1\\0&1\end{array}\right)y^{-1} = \left(\begin{array}{cc}
\frac{1}{2}&0\\
0&1\end{array}\right)\left(\begin{array}{cc}1&1\\0&1\end{array}\right) \left(\begin{array}{cc}2&0\\0&1\end{array}\right) = \left(\begin{array}{cc}
1&\frac{1}{2}\\
0&1\end{array}\right)\notin N,$$
so $N$ is not normal.
Your argument fails because in general, if $x\in X$, you do not know whether you can express $x^{-1}$ as a product of elements of $X$. You can do it if you know that for every $x\in X$, either $x$ is of finite order or else $x^{-1}\in X$. If you assume that disjunction, then your argument goes through. Alternatively, if $N$ is finite then you can conclude that $N^x=N$ (because $N^x\leq N$ and both finite of the same order gives equality), hence $N^{x^{-1}}=N$ holds as well.
There are two ways to see the condition holds in the finite case. One is that in that case, the inverses of elements of $X$ can be written as products of elements of $X$ (we are in the case where every element is of finite order). Alternatively, because $|N^x|=|N|$ and both are finite, so $N^x\leq N\implies N^x=N$ and from there you are fine.