Differentiating mahalanobis distance

Question

I would like to differentiate the mahalanobis distance:

$$D(\textbf{x}, \boldsymbol \mu, \Sigma) = (\textbf{x}-\boldsymbol \mu)^T\Sigma^{-1}(\textbf{x}-\boldsymbol \mu)$$

where $\textbf{x} = (x_1, ..., x_n) \in \mathbb R^n, \;\boldsymbol \mu = (\mu_1, ..., \mu_n) \in \mathbb R^n$ and $$\Sigma = \left( \begin{array}{ccc} E[(X_1-\mu_1)(X_1-\mu_1)] & \cdots & E[(X_1-\mu_1)(X_n-\mu_n)] \\ \vdots & \ddots & \vdots \\ E[(X_n-\mu_n)(X_1-\mu_1)] & \cdots & E[(X_n-\mu_n)(X_n-\mu_n)] \end{array} \right)$$

$\;$

is the covariance matrix. I want to differentiate $D$ with respect to $\boldsymbol\mu$ and $\Sigma$. Can someone show me how to do this? In other words, how to calculate:

$$\frac{\partial D}{\partial \boldsymbol \mu} \;\;\text{and}\;\;\frac{\partial D}{\partial \Sigma}$$? Thnx for any help!

I got the motivation for my question from this source (page 13, EM-algorithm):

http://ptgmedia.pearsoncmg.com/images/0131478249/samplechapter/0131478249_ch03.pdf

Answer 1

For convenience, define the variables $$\eqalign{ \boldsymbol{z} &= \boldsymbol{x-\mu} \cr \boldsymbol{B} &= \boldsymbol{\Sigma}^{-1} \cr } $$

and note their differentials $$\eqalign{ \boldsymbol{dz} &= \boldsymbol{dx = -d\mu} \cr \boldsymbol{dB} &= \boldsymbol{-B \cdot dB^{-1} \cdot B} \cr &= \boldsymbol{-B \cdot d\Sigma \cdot B} \cr } $$ $$ $$ Next, re-cast your objective function (taking advantage of the symmetry of $\boldsymbol B$) in terms of these variables $$\eqalign{ D &= \boldsymbol{B:zz} \cr dD &= \boldsymbol{dB:zz + 2B:z\,dz} \cr &= \boldsymbol{zz:dB + 2(B\cdot z)\cdot dz} \cr } $$

and take derivatives $$\eqalign{ \frac{\partial D}{\partial \boldsymbol z} &= \boldsymbol{0 + 2(B\cdot z)} \cr \cr \frac{\partial D}{\partial \boldsymbol B} &= \boldsymbol{zz + 0} \cr } $$ $$ $$ Now use the chain rule to revert to the original variables.

For $\boldsymbol\mu$ we have $$\eqalign{ dD &= \frac{\partial D}{\partial \boldsymbol z}\cdot \boldsymbol{dz} \cr &= \boldsymbol{2(B\cdot z)\cdot (-d\mu)} \cr \cr \frac{\partial D}{\partial \boldsymbol \mu} &= \boldsymbol{-2(B\cdot z)} \cr &= \boldsymbol{-2\Sigma^{-1}\cdot (x-\mu)} \cr } $$ $$ $$ And for $\boldsymbol\Sigma$ $$\eqalign{ dD &= \frac{\partial D}{\partial \boldsymbol B}: \boldsymbol{dB} \cr &= \boldsymbol{zz:(-B\cdot d\Sigma\cdot B)} \cr &= \boldsymbol{(-B\cdot zz\cdot B):(d\Sigma)} \cr \cr \frac{\partial D}{\partial \boldsymbol \Sigma} &= \boldsymbol{-B\cdot zz\cdot B} \cr &= \boldsymbol{-\Sigma^{-1}\cdot (x-\mu)(x-\mu)\cdot \Sigma^{-1}} \cr } $$

Answer 2

We have:

$$\eqalign{ D&=(x-\mu)^T\Sigma^{-1}(x-\mu) \cr \Sigma&=\Sigma^T \cr D&\in\mathbb{R} \cr x,\mu&\in\mathbb{R}^N \cr \Sigma&\in\mathbb{R}^{N\times N} \cr } $$

We want to find $\frac{\partial D}{\partial \mu}\in\mathbb{R}^N$ and $\frac{\partial D}{\partial \Sigma}\in\mathbb{R}^{N\times N}$, with:

$$\eqalign{ \left(\frac{\partial D}{\partial \mu}\right)_i&=\frac{\partial D}{\partial \mu_i} \cr \left(\frac{\partial D}{\partial \Sigma}\right)_{ij}&=\frac{\partial D}{\partial \Sigma_{ij}} \cr } $$

We'll start by finding $\frac{\partial D}{\partial \mu}$, by first noting that we can express $\frac{\partial D}{\partial \mu}$ using the total differential of a small change $dD$ in $D$ with respect to a small change $d\mu$ in the vector $\mu$ as follows (I'll abuse notation slightly and write $D(\mu)$ as a function of $\mu$):

$$\eqalign{ D(\mu) &= (x-\mu)^T\Sigma^{-1}(x-\mu) \\ D(\mu + d\mu) - D(\mu) =& \sum_{i=1}^N\left[ \frac{\partial D}{\partial \mu_i} d\mu_i \right] \\ =& \left( \frac{\partial D}{\partial \mu} \right)^T d\mu \\ =& (x-\mu - d\mu)^T\Sigma^{-1}(x-\mu - d\mu) - (x-\mu)^T\Sigma^{-1}(x-\mu) \\ =& (x-\mu)^T\Sigma^{-1}(x-\mu) - (x-\mu)^T\Sigma^{-1}d\mu \\ & - d\mu^T\Sigma^{-1}(x-\mu) + d\mu^T\Sigma^{-1}d\mu \\ & - (x-\mu)^T\Sigma^{-1}(x-\mu) \\ =& -2 (x-\mu)^T\Sigma^{-1}d\mu \\ =& 2 \Bigl(\Sigma^{-1} (\mu - x) \Bigr)^Td\mu \\ \Rightarrow \quad \frac{\partial D}{\partial \mu} =& 2 \Sigma^{-1} (\mu - x) }$$

To find $\frac{\partial D}{\partial \Sigma}$, we'll similarly note that the partial derivative $\frac{\partial D}{\partial A}$ of the scalar $D$ with respect to any matrix $A$ can be expressed in terms of the total differential of a small change $dD$ in $D$ with respect to a small change $dA$ in the matrix $A$, which we can express in terms of the trace operator $\mathrm{Tr}(A) = \sum_{i}{\Bigl[A_{ii}\Bigr]}$ as follows:

$$\eqalign{ dD&=\sum_{i,j}{\left[\frac{\partial D}{\partial A_{ij}}dA_{ij}\right]} \cr &=\sum_{i,j}{\left[\left(\frac{\partial D}{\partial A}\right)_{ij}dA_{ij}\right]} \cr &=\sum_{i,j}{\left[\left(\frac{\partial D}{\partial A}\right)^T_{ji}dA_{ij}\right]} \cr &=\sum_{j}{\left[\left(\frac{\partial D}{\partial A}^TdA\right)_{jj}\right]} \cr &=\mathrm{Tr}\left(\frac{\partial D}{\partial A}^TdA\right) \cr } $$

We'll now introduce the notation $A:B=\mathrm{Tr}(A^TB)$ (note that the $:$ operator can be thought of like a dot-product between two matrices, which takes two matrices as input and returns a scalar, by multiplying the two matrices element-wise and summing all the elements), and note the following identities of the trace operator:

$$\eqalign{ \mathrm{Tr}(AB) &= \sum_{i}{\left[(AB)_{ii}\right]}=\sum_{i,j}{\left[(A_{ij}B_{ji})\right]}=\sum_{i,j}{\left[B_{ji}A_{ij}\right]}=\sum_{j}{\left[(BA)_{jj}\right]}=\mathrm{Tr}(BA) \\ \mathrm{Tr}(A) &= \sum_{i}{\left[A_{ii}\right]} = \sum_{i}{\left[A^T_{ii}\right]} = \mathrm{Tr}(A^T) \\ \mathrm{Tr}(A^T B^T) &= \mathrm{Tr}((BA)^T) = \mathrm{Tr}(BA) = \mathrm{Tr}(AB) }$$

Therefore we have, for any matrix $A$:

$$\eqalign{ dD &= \frac{\partial D}{\partial A}:dA \\ } $$

Returning to $\frac{\partial D}{\partial \Sigma}$ (again I'll abuse notation slightly and write $D(\Sigma)$ as a function of $\Sigma$, and also introduce $z = x - \mu$ for brevity):

$$\eqalign{ D(\Sigma) &= z^T\Sigma^{-1}z \\ D(\Sigma + d\Sigma) - D(\Sigma) &= \frac{\partial D}{\partial \Sigma}:d\Sigma \\ &= z^T (\Sigma + d\Sigma)^{-1} z - z^T \Sigma^{-1} z \\ &= z^T \Bigl((\Sigma + d\Sigma)^{-1} - \Sigma^{-1} \Bigr) z \\ &= z^T \Bigl((\Sigma + d\Sigma)^{-1} \cdot \Sigma \cdot \Sigma^{-1} - (\Sigma + d\Sigma)^{-1} \cdot (\Sigma + d\Sigma) \cdot \Sigma^{-1} \Bigr) z \\ &= z^T (\Sigma + d\Sigma)^{-1} \cdot \Bigl(\Sigma - \Sigma - d\Sigma\Bigr) \cdot \Sigma^{-1} z \\ &= - z^T (\Sigma + d\Sigma)^{-1} \cdot d\Sigma \cdot \Sigma^{-1} z \\ &= - z^T \Sigma^{-1} \cdot d\Sigma \cdot \Sigma^{-1} z \\ &= \mathrm{Tr}\Bigl( - z^T \Sigma^{-1} \cdot d\Sigma \cdot \Sigma^{-1} z \Bigr) \\ &= \mathrm{Tr}\Bigl( - z z^T \Sigma^{-1} \cdot d\Sigma \cdot \Sigma^{-1} \Bigr) \\ &= \mathrm{Tr}\Bigl( - \Sigma^{-1} z z^T \Sigma^{-1} \cdot d\Sigma \Bigr) \\ &= - \Sigma^{-1} z z^T \Sigma^{-1} : d\Sigma \\ \Rightarrow \quad \frac{\partial D}{\partial \Sigma} &= - \Sigma^{-1} z z^T \Sigma^{-1} } $$