I think that the criterion that you allude to is the following:
Assume that $X^p - a$ has no roots in $\Bbb F _{p^n}$. Then $X^{p^m} - a$ is irreducible in $\Bbb F _{p^n}[X]$ $\forall m \ge 1$.
In any case, you can't use it here.
1) Let us see how to prove that the second polynomial (call it $P$) is reducible. First, none of $0, 1, 2$ is a root, so $P$ has no factor of degree $1$ (and, therefore, no factor of degree $4$). Let us see whether $P$ can be written as a product of two polynomials, of degree $2$ and, respectively, $3$. The long method is to multiply the polynomials $aX^2 + bX + c$ and $eX^3 + fX^2 + gX + h$, equate the coefficients of equal powers etc...
A shorter approach is the following: let us list all the polynomials of degree $2$ and check whether they divide $P$. Since $P$ has no linear factor, it is enough to list only those polynomials of degree $2$ that are irreducible. Note that no such polynomial may end in $0$, so the constant term is $1$ or $2$. Concerning the leading coefficient, it is enough to consider only monic polynomials. So far we get the polynomials $X^2 + aX + 1$, $X^2 + aX +2$. A polynomial of degree $2$ is irreducible if it has no roots, i.e. if its discriminant is not a perfect square. Since the only perfect squares in $\Bbb F _3$ are $0$ and $1$, you want $a$ such that the discriminant should be $2$.
In the first case, the discriminant is $a^2 - 4$, so you want $a$ such that $a^2 - 4 = 2$, so $a=0$.
In the second case, the discriminant is $a^2 - 8$, so you want $a$ such that $a^2 - 8 = 2$, i.e. $a^2 = 1$, i.e. $a=1$ or $a=2$.
So, the only monic irreducible polynomials of degree $2$ are $X^2 + 1$, $X^2 + X + 2$, $X^2 + 2X +2$. Let us see which one divides our polynomial.
Note that $P = X^3 (X^2+1) + X^2 (X^2+1) + X-1$, so when you divide $P$ by $X^2 +1$ you get the remainder $X-1$, so $X^2-1 \not| P$.
Finally, try to divide $P$ by the last two polynomials. $X^2 + 2X +2$ will turn out to be a factor.
2) Concerning the first polynomial (call it $Q$), the approach will be similar. First, note that it has no roots, so it has no linear factor. Therefore, we are going to look only for irreducible factors of degree $2, \dots, 5$. In order to be irreducible, these potential factors must have the constant term $1$.
Looking for irreducible polynomials of degree $2$, these must look like $X^2 +aX +1$. Clearly, $a=1$ gives the only irreducible one.
For degree $3$, you want those polynomials $X^3 + aX^2 + bX +1$ that have no linear factor; since $0$ cannot be a root, you also do not want $1$ to be so, therefore you want $1+a+b+1 \ne 0$, which means $a+b =1$, so the only possibilities are $X^3 + X^2 +1$ and $X^3 +X+1$.
In degree $4$, you want those polynomials $X^4 + aX^3 + bX^2 + cX +1$ that have no roots (so $1+a+b+c+1 \ne 0$, i.e. $a+b+c=1$) and that have no irreducible factor of degree $2$, i.e. that are not divided by $X^2+X+1$ (found above). A reducible factor of degree $4$ having no root would have to be $(X^2+X+1)^2 = X^4 + X^2 +1$. Therefore, the only irreducible polynomials of degree $4$ remain $X^4 + X^3 +1$, $X^4+ X+1$ and $X^4+ X^3 + X^2 + X + 1$.
Finally, the reducible polynomials $x^5 + aX^4 + bX^3 +cX^2 + dX +1$ of degree $5$ are those that have roots (i.e. $a+b+c+d=0$) and those that can be divided by $X^2+1$. Performing long division by $X^2+1$, you get the remainder $(b+d+1)x + (a+c+1)$, so in order to get the reducible polynomials impose $a+b+c+d = 0, \; b+d+1 = 0, \; a+c+1 = 0$. Solve this system (it will have several solutions); the polynomials that are not among these solutions are the irreducible ones of degree $5$.
Now that you've listed all the irreducible polynomials of degree $\le 5$, check (by performing long division or by computing the greatest common divisor) which ones divide $Q$. None will, so $Q$ is irreducible.
Below is the proof of the irreducibility criterion mentioned at the beginning of my post.
Notice that $X^{p^m} - a$ has at least one root $x$ in some algebraic closure $K$ of $\mathrm F_{p^m}$; if $y \in K$ is another root, it follows that $x^{p^m} = y^{p^m}$ and, since $r \mapsto r^{p^m}$ is an automorphism of $K$ (because the Frobenius map $r \mapsto r^p$ is), it follows that $x=y$. It follows that $X^{p^m} - a$ has exactly one root $x \in K$, of multiplicity $p^m$.
If $g \in \mathrm F_{p^m} [X]$ is the minimal polynomial of $x$, then $X^{p^m} - a = g^s$; since $p^m = s \deg g$, it follows that $s = p^t$. Let $b = -g(0)$ and assume $t>0$. Evaluating $X^{p^m} - a = g^s$ in $0$, and assuming $t>0$, we get $a = (b^{p^{t-1}})^p$ (because $-1 = (-1)^s$ in characteristic $p>0$), which would imply that $X^p - a$ has the root $b^{p^{t-1}} \in \mathrm F _{p^m}$, which would contradict the hypothesis of the criterion. It follows that $t=0$, so that $s=1$, therefore $X^{p^m} - a$ is the minimal polynomial of $a$, therefore irreducible by the definition of the concept of "minimal polynomial".