What is the derivative of the $(m,n)^{th}$ element of the inverse, $X^{-1}$, with respect to the $(i,j)^{th}$ element of $X$?
i.e. what is;
$$\frac{\partial X^{-1}_{mn}}{\partial X_{ij}}$$
Is it true that;
$$\frac{\partial A^{-1}_{mn}}{\partial A_{ij}}=-\sum_k \sum_l A^{-1}_{mk} \frac{\partial A_{kl}}{\partial A_{ij}} A^{-1}_{ln}=-A^{-1}_{mi}A^{-1}_{jn}$$
$ \def\p{\partial}\def\X{X^{-1}} \def\L{\left}\def\R{\right}\def\LR#1{\L(#1\R)} \def\grad#1#2{\frac{\p #1}{\p #2}} $Let $e_k$ denote a standard basis vector, whose components are all zero except for the $k^{th}$ element which is equal to ${\tt1}.\,$ Such vectors can be used to write the component-wise self-gradient of a matrix $$\eqalign{ \grad{X}{X_{ij}} &= e_i\,e_j^T \\ }$$ The component-wise gradient of the inverse can be calculated as $$\eqalign{ I &= X\X \\ \grad{I}{X_{ij}} &= \LR{\grad{X}{X_{ij}}}\X + X\LR{\grad{\X}{X_{ij}}} \\ X\LR{\grad{\X}{X_{ij}}} &= -\LR{\grad{X}{X_{ij}}}\X \\ \grad{\X}{X_{ij}} &= -\X\LR{e_i\,e_j^T}\X \\ }$$ To extract the $(m,n)$ component multiply by more basis vectors $$\eqalign{ \grad{\X_{mn}}{X_{ij}} &= e_m^T\LR{\grad{\X}{X_{ij}}}e_n \\ &= -e_m^T\X e_i\;e_j^T\X e_n \\ &= -\X_{mi}\;\X_{jn} \\ }$$ which confirms the formula.
Another related approach is as follows. Consider the Frobenius scalar product $\phi=\mathbf{e}_m : \mathbf{X}^{-1} \mathbf{e}_n$
The differential writes \begin{eqnarray} d\phi &=& -\mathbf{e}_m: \mathbf{X}^{-1} (d\mathbf{X}) \mathbf{X}^{-1} \mathbf{e}_n \\ &=& - \mathbf{X}^{-T} \mathbf{e}_m \mathbf{e}_n^T \mathbf{X}^{-T} : d\mathbf{X} \\ \end{eqnarray} The LHS term in the scalar product is the (whole) matrix gradient.
If you are looking for the $(i,j)$-th component, the requested quantity is $- (\mathbf{e}_i^T \mathbf{X}^{-T} \mathbf{e}_m) (\mathbf{e}_n^T \mathbf{X}^{-T} \mathbf{e}_j)$ which is exactly what you obtained.
