Let $K=\{-k_1,\ldots,-k_n\}$.
Suppose the given polynomial $P(x)$ is reducible. We may then write $P(x) = Q(x)R(x)$ for some non-constant monic polynomials $Q,R \in \mathbb{Z}[x]$ of degree less than $n$.
Note then that $Q(k)R(k)=1$ for all $k \in K$. Now, the only ways to write 1 as the product of 2 integers are $1=(1)(1)$ and $1=(-1)(-1)$. This implies that $Q(k)=R(k)$ for all $k \in K$. The only way that two polynomials of degree less than $n$ can be identical at $n$ distinct points is if they are identical. So $Q=R$, and we may write $P(x)=Q(x)^2$.
Define $K(c)$ to be the set of $k\in K$ for which $Q(k)=c$, so that $K(+1)$ and $K(-1)$ form a partition of $K$. We may then write, for $c\in \{\pm 1\}$,
$$
Q(x) = c + Q_c(x)\prod_{k \in K(c)} (x-k)
$$
for some monic polynomials $Q_c(x)$. In fact, it is easy to see that $Q_c(x)=1$, since squaring $Q(x)$ yields a polynomial whose degree is $\deg(Q_1) + \deg(Q_{-1}) + |K(1)| + |K(-1)|$. For this to equal the degree-$n$ polynomial $P(x)$ demands that $Q_1$ and $Q_{-1}$ are constant polynomials, since $K(1)$ and $K(-1)$ are a partition of the size-$n$ set $K$. It also follows that $|K(1)|=|K(-1)|$.
So we have
$$
Q(x) = c + \prod_{k \in K(c)} (x-k)
$$
for $c \in \{\pm1\}$, which yields the following factorizations of $\pm2$:
$$
\begin{align}
+2 &= \prod_{k \in K(-1)} (x-k) & \mathrm{ for }\ x\in K(1) \\
-2 &= \prod_{k \in K(1)} (x-k) & \mathrm{ for }\ x\in K(-1)
\end{align}
$$
Let $m=|K(1)|$. Let $a_1 < \cdots < a_m$ be the elements of $K(1)$, and let $b_1<\cdots<b_m$ be the elements of $K(-1)$.
Plugging $x=a_1$ into the first equation, we find that the set $\{a_1-b_i\}_{1\leq i \leq m}$ is a factorization of $+2$. We can enumerate the possible values of this set using the fact that 2 can only be written as the product of distinct integers in 4 ways: $2 = (2) = (-2)(-1) = (1)(2) = (-2)(-1)(1)$.
Case 1: $\{2\}$
Then $m=1$ and $a_1=b_1+2$. This yields the OP's $n=2$ solution.
Case 2: $\{-2,-1\}$
Then $m=2$ and $a_1 = b_1-1=b_2-2$. Setting $x=a_2$ in the factorization of $+2$ yields $a_2=a_1+3$. This yields the OP's $n=4$ solution.
Case 3: $\{1,2\}$
Similar to Case 2.
Case 4: $\{-2,-1,1\}$
Then $m=3$, and we have that each of $\pm2$ has 3 distinct factorizations of size 3, which clearly is impossible.