The so-called Morales–Ramis theory seems to be what you are looking for. Several articles, plus a draft version of the book Differential Galois Theory and Non-integrability of Hamiltonian Systems can be downloaded from the homepage of Prof. Juan J. Morales-Ruiz.
I can't say that I know very much about this subject, but the little that I do know, I have learned from Maria Przybylska, who has written many papers together with Andrzej Maciejewski where they apply this theory. For example, they have found some polynomial potentials $V(x,y)$ such that the Hamiltonian system generated by $H=\frac12(p_x^2+p_y^2)+V(x,y)$ has an additional integral of motion (independent of $H$) which is of degree four in the momenta $p_x$ and $p_y$. Even though the polynomial $V(x,y)$ can be simple-looking, the second integral of motion may be completely monstrous (many pages of Mathematica output), and it's nothing that you would ever stumble upon just by blind trial and error, but using Morales–Ramis theory it's possible to systematically rule out non-integrable cases and then find these rare and very special integrable potentials.