First, some facts about numbers in your sequence. Let $n=x^2-x+1$. If $x=3k-1$, then
\begin{align*}
n=x^2-x+1 &= (3k-1)^2-(3k-1)+1\\
&= 9k^2-6k+1-3k+1+1\\
&= 9k^2-9k+3\\
\implies n/3 &= 3k^2-3k+1
\end{align*}
$k^2$ and $k$ have the same parity. Therefore
$n/3=1\mod 6$. If $x$ is any other integer then $n=1\mod 6$ as can be easily checked. Thus $n$ is odd; and, after a factor of 3 has been extracted, if there is one, the result is 1, modulo 6. In any case $n\ne 5\mod 6$.
Suppose your conjecture is wrong, and let $p$ be the lowest prime counterexample, i.e. lowest prime $p\ne3$, not of the form $6k+1$, which divides a number that is in your sequence.
Let prime $p$ divide $n=x^2-x+1$. Then $p\ne 2$. Because $p\ne3$ and $p\ne1\mod 6$, $p=5\mod 6$.
Let $x=sp+r$ where $s$ is the nearest integer to $x/p$, so $|r|<p/2$. Then
\begin{align*}
n=x^2-x+1 &= (sp+r)^2-(sp+r)+1\\
&= s^2p^2+2srp+r^2-sp-r+1\\
&= Ap+r^2-r+1
\end{align*}
where $A=s^2p+2sr-s$. Thus
$p\mid r^2-r+1$, so, for some integer $m'$,
\begin{align*}
m'p &= r^2-r+1\\
&\leqslant r^2\\
&<p^2/4\\
\implies m' &< p/4.
\end{align*}
If $3\mid m'$, let $m=m'/3$, otherwise let $m=m'$. Then $3\nmid m$, and $m\leqslant m'<p/4$.
Because $m<p$ and $p$ is the smallest counterexample, each prime factor $q$ of $m$ is 1, modulo 6 (because $q\leqslant m<p$ so $q$ is not a counterexample). So, modulo 6, $m=1$, $p=5$, so $mp=5$. But $mp = r^2-r+1$ is in your sequence, so $mp\ne 5\mod 6$. This is a contradiction. (In fact, for any integer $m$ where $m=5\mod 6$, at least one prime factor of $m$ is $q=5\mod 6$.)
Therefore your conjecture is correct.