If $\lambda$ is an eigenvalue of $A$, then $1+\lambda$ is an eigenvalue of $I+A$. Indeed, let $v$ be an eigenvector to the eigenvalue $\lambda$, i.e. $Av = \lambda v$. Then $(I+A)v = v + \lambda v = (1+\lambda)v$.
But then $\det(I+A) = \prod_{i} (1+\lambda_i)$, where the product is over all eigenvalues $\lambda_i$ of $A$ (and the same factors can turn up multiple times).
So $I+A$ is invertible if and only if $\lambda_i \ne -1$ for all $\lambda_i$.
Note that the above is the same as your criterion for $2\times 2$ matrices:
$$\prod_{i} (1+\lambda_i) = 1 + (\lambda_1 + \lambda_2) + \lambda_1 \lambda_2 = 1 + \mathrm{tr} A + \det A$$