Computing the $4 \times 4$ matrix would certainly work, but for large enough matrices (and I would consider $4 \times 4$ just large enough), it is quicker to row reduce the matrix. The matrix is invertible if and only if you can row reduce the matrix down to an upper triangular matrix (row-echelon form) with non-zero entries on the diagonal. In other words, there is a pivot in each column.
As an aside that you don't need to understand at the moment, with a bit more of an advanced eye, you can quickly see that this matrix is indeed invertible, as it is $J - I$, where $J$ is the matrix of $1$s. The matrix of $1$s has eigenvalues $0$ and $4$ ($0$ and $n$ more generally, where we are working with $n \times n$ matrices), see here. This means the eigenvalues of $J - I$ are $-1$ and $3$, which doesn't include $0$, the matrix is invertible.