Let $b$ be a column vector, $A$ a symmetric square matrix and $X$ a square matrix. Define the scalar $$ f(X) = b^T(I-X)^{-1}A((I-X)^{-1})^Tb. $$ I would like to find an expression for $\frac{dF}{dX}$. Note that this should be a matrix.
The question derivative of inverse matrix by itself provides an answer for a simpler expression, but I am not able to adapt the derivations to the current setup.
Introduce the matrix variable
$$\eqalign{
\def\c#1{\color{red}{#1}}
\def\qiq{\quad\implies\quad}
Y &= (I-X)^{-1} \qiq \c{dY=Y\:dX\;Y} \\
}$$
the sym() function
$${\rm sym}(A) = \frac12(A+A^T) \qquad\qquad\qquad\;$$
and the matrix inner product $(:)$
$$\eqalign{
A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; {\rm Tr}(A^TB) \\
A:A &= \|A\|^2_F \qquad \{ {\rm Frobenius\;norm} \} \\\\
}$$
Use the above notation to write the function,
then calculate its differential and gradient
$$\eqalign{
f &= bb^T:YAY^T \\
df &= bb^T:2\:{\rm sym}(dY\,AY^T) \\
&= 2bb^T:(\c{dY}\,AY^T) \\
&= 2bb^TYA:(\c{Y\,dX\:Y}) \\
&= 2Y^Tbb^TYAY^T:dX \\
\frac{\partial f}{\partial X} &= 2Y^Tbb^TYAY^T \\
}$$
The differential of $X\mapsto X^{-1}$ is the map $H\mapsto -X^{-1}HX^{-1}.$ Proof: $$(X+H)^{-1}-X^{-1}=X^{-1}((I+HX^{-1})^{-1}-I)=X^{-1}(I-HX^{-1}-o(H)-I)$$
