The fact that the two Stirling matrices are inverses is a special case of the fact that certain matrices consisting of elementary and complete symmetric polynomials are inverses.
Define infinite lower triangular matrices $F$ and $G$ consisting of elementary and complete symmetric polynomials, respectively, via
$$(F(z_1, z_2, \ldots ))_{ij} = e_{i-j}(z_1, z_2, \ldots, z_{i-1}),$$
$$(G(z_1, z_2, \ldots ))_{ij} = h_{i-j}(z_1, z_2, \ldots, z_j).$$
For example, if $[F]_n$ and $[G]_n$ denote the $n \times n$ first principal minors of $F$ and $G$, respectively, then
$$[F(x,y,z)]_4 = \begin{bmatrix}
1 & 0 & 0 & 0 \\
x & 1 & 0 & 0 \\
xy & x+y & 1 & 0 \\
xyz & xy+xz+yz & x+y+z & 1
\end{bmatrix}$$
and
$$
[G(x,y,z)]_4 = \begin{bmatrix}
1 & 0 & 0 & 0 \\
x & 1 & 0 & 0 \\
x^2 & x+y & 1 & 0 \\
x^3 & x^2 + xy + y^2 & x+y+z & 1
\end{bmatrix}.$$
Then $F(1,1,\ldots)$ is the infinite Pascal matrix (consisting of the binomial coefficients), and $F(-1,-2,-3,\ldots)$ is the infinite Stirling matrix of the first kind, so that
$$[F(1,1,\ldots)]_4 = \begin{bmatrix}
1 & 0 & 0 & 0 \\
1 & 1 & 0 & 0 \\
1 & 2 & 1 & 0 \\
1 & 3 & 3 & 1
\end{bmatrix}, \text{ and }
[F(-1,-2,-3,\ldots)]_4 = \begin{bmatrix}
1 & 0 & 0 & 0 \\
-1 & 1 & 0 & 0 \\
2 & -3 & 1 & 0 \\
-6 & 11 & -6 & 1
\end{bmatrix}.$$
Moreover, $G(1,1,\ldots)$ is also the infinite Pascal matrix, and $G(1,2,3,\ldots)$ is the infinite Stirling matrix of the second kind, so that, for example,
$$[G(1,1,\ldots)]_4 = \begin{bmatrix}
1 & 0 & 0 & 0 \\
1 & 1 & 0 & 0 \\
1 & 2 & 1 & 0 \\
1 & 3 & 3 & 1
\end{bmatrix}, \text{ and }
[G(1,2,3,\ldots)]_4 = \begin{bmatrix}
1 & 0 & 0 & 0 \\
1 & 1 & 0 & 0 \\
1 & 3 & 1 & 0 \\
1 & 7 & 6 & 1
\end{bmatrix}.$$
In "Symmetric Polynomials, Pascal Matrices, and Stirling Matrices" (Linear Algebra and Its Applications, 428 (4): 1127-1134, 2008) Andy Zimmer and I prove that $$F(z_1,z_2,\ldots) G(-z_1, -z_2, \ldots) = I.$$
Thus the two Stirling matrices are inverse, and (up to sign) the Pascal matrix is its own inverse.
(The fact that the binomial coefficients and the Stirling numbers can be expressed in terms of symmetric polynomials is known; for references, see the paper.)