By the rules of Pascal's Triangle,
$$
\binom{i+1+j}{j}-\binom{i+j}{j}=\binom{i+j}{j-1}
$$
which means that row $i+1$ minus row $i$ gives row $i+1$ shifted to the right.
$$
a_{i+1,j}-a_{i,j}=a_{i+1,j-1}
$$
This can be done to shift row $n$ to the right, then shift row $n-1$, up to row $2$. We can repeat the process to shift rows $n$ through $3$ to the right. We can continue this until we have all the $1$s on the diagonal and $0$s in the lower left triangle.
Subtracting one row from another does not change the determinant, so the original determinant was $1$.
Using Cramer's Rule, we get that the inverse is an integer matrix.
Subtract row $3$ from row $4$:
$$\small
\begin{bmatrix}
\binom{0}{0}&\binom{1}{1}&\binom{2}{2}&\binom{3}{3}\\
\binom{1}{0}&\binom{2}{1}&\binom{3}{2}&\binom{4}{3}\\
\binom{2}{0}&\binom{3}{1}&\binom{4}{2}&\binom{5}{3}\\
\binom{3}{0}&\binom{4}{1}&\binom{5}{2}&\binom{6}{3}\\
\end{bmatrix}
\rightarrow
\begin{bmatrix}
\binom{0}{0}&\binom{1}{1}&\binom{2}{2}&\binom{3}{3}\\
\binom{1}{0}&\binom{2}{1}&\binom{3}{2}&\binom{4}{3}\\
\binom{2}{0}&\binom{3}{1}&\binom{4}{2}&\binom{5}{3}\\
0&\binom{3}{0}&\binom{4}{1}&\binom{5}{2}\\
\end{bmatrix}
$$
Subtract row $2$ from row $3$:
$$\small
\begin{bmatrix}
\binom{0}{0}&\binom{1}{1}&\binom{2}{2}&\binom{3}{3}\\
\binom{1}{0}&\binom{2}{1}&\binom{3}{2}&\binom{4}{3}\\
\binom{2}{0}&\binom{3}{1}&\binom{4}{2}&\binom{5}{3}\\
0&\binom{3}{0}&\binom{4}{1}&\binom{5}{2}\\
\end{bmatrix}
\rightarrow
\begin{bmatrix}
\binom{0}{0}&\binom{1}{1}&\binom{2}{2}&\binom{3}{3}\\
\binom{1}{0}&\binom{2}{1}&\binom{3}{2}&\binom{4}{3}\\
0&\binom{2}{0}&\binom{3}{1}&\binom{4}{2}\\
0&\binom{3}{0}&\binom{4}{1}&\binom{5}{2}\\
\end{bmatrix}
$$
Subtract row $1$ from row $2$:
$$\small
\begin{bmatrix}
\binom{0}{0}&\binom{1}{1}&\binom{2}{2}&\binom{3}{3}\\
\binom{1}{0}&\binom{2}{1}&\binom{3}{2}&\binom{4}{3}\\
0&\binom{2}{0}&\binom{3}{1}&\binom{4}{2}\\
0&\binom{3}{0}&\binom{4}{1}&\binom{5}{2}\\
\end{bmatrix}
\rightarrow
\begin{bmatrix}
\binom{0}{0}&\binom{1}{1}&\binom{2}{2}&\binom{3}{3}\\
0&\binom{1}{0}&\binom{2}{1}&\binom{3}{2}\\
0&\binom{2}{0}&\binom{3}{1}&\binom{4}{2}\\
0&\binom{3}{0}&\binom{4}{1}&\binom{5}{2}\\
\end{bmatrix}
$$
Subtract row $3$ from row $4$:
$$\small
\begin{bmatrix}
\binom{0}{0}&\binom{1}{1}&\binom{2}{2}&\binom{3}{3}\\
0&\binom{1}{0}&\binom{2}{1}&\binom{3}{2}\\
0&\binom{2}{0}&\binom{3}{1}&\binom{4}{2}\\
0&\binom{3}{0}&\binom{4}{1}&\binom{5}{2}\\
\end{bmatrix}
\rightarrow
\begin{bmatrix}
\binom{0}{0}&\binom{1}{1}&\binom{2}{2}&\binom{3}{3}\\
0&\binom{1}{0}&\binom{2}{1}&\binom{3}{2}\\
0&\binom{2}{0}&\binom{3}{1}&\binom{4}{2}\\
0&0&\binom{3}{0}&\binom{4}{1}\\
\end{bmatrix}
$$
Subtract row $2$ from row $3$:
$$\small
\begin{bmatrix}
\binom{0}{0}&\binom{1}{1}&\binom{2}{2}&\binom{3}{3}\\
0&\binom{1}{0}&\binom{2}{1}&\binom{3}{2}\\
0&\binom{2}{0}&\binom{3}{1}&\binom{4}{2}\\
0&0&\binom{3}{0}&\binom{4}{1}\\
\end{bmatrix}
\rightarrow
\begin{bmatrix}
\binom{0}{0}&\binom{1}{1}&\binom{2}{2}&\binom{3}{3}\\
0&\binom{1}{0}&\binom{2}{1}&\binom{3}{2}\\
0&0&\binom{2}{0}&\binom{3}{1}\\
0&0&\binom{3}{0}&\binom{4}{1}\\
\end{bmatrix}
$$
Subtract row $3$ from row $4$:
$$\small
\begin{bmatrix}
\binom{0}{0}&\binom{1}{1}&\binom{2}{2}&\binom{3}{3}\\
0&\binom{1}{0}&\binom{2}{1}&\binom{3}{2}\\
0&0&\binom{2}{0}&\binom{3}{1}\\
0&0&\binom{3}{0}&\binom{4}{1}\\
\end{bmatrix}
\rightarrow
\begin{bmatrix}
\binom{0}{0}&\binom{1}{1}&\binom{2}{2}&\binom{3}{3}\\
0&\binom{1}{0}&\binom{2}{1}&\binom{3}{2}\\
0&0&\binom{2}{0}&\binom{3}{1}\\
0&0&0&\binom{3}{0}\\
\end{bmatrix}
$$
The determinant is $\binom{0}{0}\binom{1}{0}\binom{2}{0}\binom{3}{0}=1$.