If you write down $(A-I)^2$ you see that any vector with the final entry nonzero is outside the kernel, so you can pick
$$
p_4 =
\left(
\begin{array}{c}
0 \\
0 \\
0 \\
1
\end{array}
\right)
$$
followed by $p_3 = (A-I) p_4 \;, \;$ $p_2 = (A-I) p_3 \;, \;$ which is automatically an eigenvector because $(A-I)p_2 = (A-I)^3 p_4 = 0.$
Then let the first column $p_1$ be a different eigenvector from $p_2.$ Notice that, with two eigenvectors available, I deliberately chose a linear combination so the entires of the first column would be small. I hoped it would reduce the determinant of $P,$ and this worked out nicely, the determinant came out to $2.$ You can see this is the factor of $\frac{1}{2}$ required to write all the matrices with integers.
$$
\frac{1}{2}
\left(
\begin{array}{cccc}
1&5&3&0 \\
-9&-35&-35&0 \\
2&8&8&0 \\
0&0&0&2
\end{array}
\right)
\left(
\begin{array}{cccc}
0&-8&-35&0 \\
1&1&4&0 \\
-1&1&5&0 \\
0&0&0&1
\end{array}
\right) =
\left(
\begin{array}{cccc}
1&0&0&0 \\
0&1&0&0 \\
0&0&1&0 \\
0&0&0&1
\end{array}
\right)
$$
$$
\frac{1}{2}
\left(
\begin{array}{cccc}
1&5&3&0 \\
-9&-35&-35&0 \\
2&8&8&0 \\
0&0&0&2
\end{array}
\right)
\left(
\begin{array}{cccc}
-7&-32&-32&-35 \\
1&5&4&4 \\
1&4&5&5 \\
0&0&0&1
\end{array}
\right)
\left(
\begin{array}{cccc}
0&-8&-35&0 \\
1&1&4&0 \\
-1&1&5&0 \\
0&0&0&1
\end{array}
\right) =
\left(
\begin{array}{cccc}
1&0&0&0 \\
0&1&1&0 \\
0&0&1&1 \\
0&0&0&1
\end{array}
\right)
$$
and
$$
\frac{1}{2}
\left(
\begin{array}{cccc}
0&-8&-35&0 \\
1&1&4&0 \\
-1&1&5&0 \\
0&0&0&1
\end{array}
\right)
\left(
\begin{array}{cccc}
1&0&0&0 \\
0&1&1&0 \\
0&0&1&1 \\
0&0&0&1
\end{array}
\right)
\left(
\begin{array}{cccc}
1&5&3&0 \\
-9&-35&-35&0 \\
2&8&8&0 \\
0&0&0&2
\end{array}
\right) =
\left(
\begin{array}{cccc}
-7&-32&-32&-35 \\
1&5&4&4 \\
1&4&5&5 \\
0&0&0&1
\end{array}
\right)
$$