$$A=\left(\begin{array}{cccc}0&0&0&1\\ 1&0&0&0\\ 0&1&0&0\\ 0&0&1&0\end{array}\right).$$
Finding a matching $B$ is left as an exercise to the reader with the hint: the permutation $(1234)$ and its inverse $(4321)$ are conjugates in $S_4$.
That matrix does have $\lambda=1$ as an eigenvalue, so it is conjugate to a matrix in $GL_1\times GL_3$. Because $\lambda=-1$ is also an eigenvalue, it also stabilizes two 2-dimensional subspaces over $\Bbb{Q}$ and is thus also conjugate to a matrix from $GL_2\times GL_2$. Apparently this is undesirable.
Matching the updated version of the question:
The palindromic polynomial
$$f(x)=x^4+x^3+3x^2+x+1$$
is irreducible over $\Bbb{Z}$ because it remains irreducible after reduction modulo two. Because it is palindromic, $x^4 f(1/x)=f(x)$, its zeros have the property that $f(\rho)=0$ if and only if $f(1/\rho)=0$. Its companion matrix is
$$
A=\left(
\begin{array}{cccc}
0 & 0 & 0 & -1 \\
1 & 0 & 0 & -1 \\
0 & 1 & 0 & -3 \\
0 & 0 & 1 & -1 \\
\end{array}
\right),
$$
when the inverse is
$$
A^{-1}=\left(
\begin{array}{cccc}
-1 & 1 & 0 & 0 \\
-3 & 0 & 1 & 0 \\
-1 & 0 & 0 & 1 \\
-1 & 0 & 0 & 0 \\
\end{array}
\right).$$
The palindromic property of $f(x)$ then tells us that $A$ and $A^{-1}$ share the same set of simple eigenvalues. This is because the eigenvalues of a companion matrix $A$ are the roots of the polynomial $f(x)$. Furthermore, the eigenvalues of $A^{-1}$ are the reciprocals of the eigenvalues of $A$, which in this case are still zeros of $f(x)$, and thus the eigenvalues of $A$.
The shared simple eigenvalues already mean that the two matrices are automatically conjugate in $GL_4(\Bbb{C})$, but it turns out that they are also conjugate in $GL_4(\Bbb{Z})$.
Unless I made a mistake we can use the same $B$ as above, namely
$$B=\left(
\begin{array}{cccc}
0 & 0 & 0 & 1 \\
0 & 0 & 1 & 0 \\
0 & 1 & 0 & 0 \\
1 & 0 & 0 & 0 \\
\end{array}
\right).
$$
Irreducibility of $f(x)$ translates into the non-existence of a non-trivial subspace $V\subset \Bbb{Q}^4$ such that $AV=V$. This rules out the possibility of $A$ being conjugate to a matrix from $GL_1(\Bbb{Q})\times GL_3(\Bbb{Q})$ or $GL_2(\Bbb{Q})\times GL_2(\Bbb{Q})$.
The same procedure works for any palindromic quartic. So it is also possible to arrange all the eigenvalues to be real. This happens for example with the companion matrix of
$$g(x)=x^4 - 5 x^3 - 13 x^2 - 5 x + 1.$$