In the standard abstract algebra curriculum, one learns a battery of irreducibility tests for factoring polynomials over $\mathbb{Z}$ (equivalently, by Gauss' lemma, over $\mathbb{Q}$). For instance (not all names standard):
- Linear Factor Test: A polynomial has a linear factor over $\mathbb{Z}$ if and only if it has a root in $\mathbb{Q}$.
- Quadratic/Cubic Test: A polynomial of degree 2 or 3 is reducible if and only if it has a linear factor.
- Brute Force Method: Write out the forms of all possible factorizations. For instance, after checking a quartic for linear factors, look at $(X^2+aX+b)(X^2+cX+d)$. Obtain a system of equations for the coefficients. Determine whether solutions exist. Ugh. (Though, more feasible over $\mathbb{Z}_p$.)
- Mod-$p$ Irreducibility Test: If there exists a prime $p$ such that a polynomial is irreducible over $\mathbb{Z}_p$, then it is irreducible over $\mathbb{Z}$.
- Eisenstein's Criterion: If there exists a prime $p$ which divides all but the lead coefficient, and whose square does not divide the constant term, then the polynomial is irreducible.
- Substitution tricks: The reducibility of a given polynomial $f(X)$ is related to the reducibility of other polynomials like $f(aX+b)$ or the reversal $X^n f(1/X)$.
- Complexify: Factor the polynomial into linear factors over $\mathbb{C}$. Every higher-degree divisor of the polynomial is a product of several of these linear factors. Try out all the products of the linear factors and verify that all of them have non-integer coefficients. (See for instance Jyrki Lahtonen's solution in this post).
- Special cases: E.g. cyclotomic polynomials are something you should just know.
These are often used in combination. One can prove that $X^4+X+1$ is irreducible over $\mathbb{Z}$ by showing it's irreducible over $\mathbb{Z}_2$, which in turn can be easily done by the "brute-force" approach since there are very few quadratics over $\mathbb{Z}_2$. Jyrki Lahtonen's solution in this post shows that $f(X):=X^4-10X^2+1$ is irreducible by applying Eisenstein with $p=2$ to the reversal of $\frac{1}{8} f(2X+1)$. (Gorgeous!)
How does one get a sense for which tricks to try when? There are infinitely many primes $p$ to try with Eisenstein and mod-$p$ tests, though in practice $p$ tends to be small. Allowing substitution tricks opens up a dizzying array of possibilities. I'm thinking by analogy with the convergence tests one learns in calculus. One can just try different approaches until one of them works, but one can also see patterns: a series with powers or factorials is likely amenable to the Ratio Test, terms of "smaller order" can be eliminated by the Limit Comparison Test, terms that you know how to bound can be handled by the Comparison Test, and functions you know how to integrate are promising candidates for the Integral Test.
Are there analogous clues to look for here? How might you "smell" which test is likely to work with which polynomial?