3

I'm reading the Deep Learning book and I've encountered a section that is difficult for me to understand. Specifically transformation from equation (5.38) to (5.39) $$ \mathbb{E}\left[\hat{\sigma}^2_m\right]=\mathbb{E}\left[\frac{1}{m}\sum_{i=1}^{m}\left( x^{(i)} - \hat{\mu}_m \right)^2 \right] \\ = \frac{m - 1}{m}\sigma^2 $$

Can anyone explain me how we got to the $\frac{m - 1}{m}\sigma^2$?

Tomasz Bekas
  • 133

2 Answers2

2

If you expand the square you have \begin{align*} \mathbb{E} \frac{1}{m} \sum_i \left( x_i^2 - 2 x_i \hat{\mu} + \hat{\mu}^2 \right) &= \frac{1}{m} \mathbb{E} \sum_i x_i^2 - \mathbb{E} \left[ 2\hat{\mu} \frac{1}{m}\sum_i x_i + \hat{\mu}^2 \right] \\ &= \mathbb{E} \left[ X^2 \right] - \mathbb{E} \left[ \hat{\mu}^2 \right], \end{align*} Now we have \begin{align*} \left( \sum_i x_i \right)^2 = m \sum_i x_i^2 - \frac{1}{2}\sum_i \sum_j (x_i - x_j)^2 \end{align*} Taking care of those points where $i = j$ gives \begin{align*} \mathbb{E}\left[ \left( \sum_i x_i \right)^2\right] &= m^2 \mathbb{E}\left[ X^2 \right] - \frac{1}{2}\sum_i \sum_{j \neq i} \mathbb{E} \left( x_i^2 - 2x_i x_j + x_j^2 \right) \\ &= m^2 \mathbb{E}\left[ X^2 \right] - m(m-1)\left( \mathbb{E}[X^2] - \mathbb{E}[X]^2 \right) \\ &= m^2 \mathbb{E}[X^2] - m(m-1)\sigma^2, \end{align*} so that after dividing through by $m^2$ we get \begin{align*} \mathbb{E} \left[ X^2 \right] - \mathbb{E}\left[ \hat{\mu}^2 \right] = \frac{(m-1)}{m}\sigma^2 \end{align*}

Nadiels
  • 2,648
  • Thank you for the answer, I don't understand it quite yet. Could you explain me how did we get $\mathbb{E}\left[X^2\right]$ from $\frac{1}{m}\mathbb{E}\sum_{i}x_i^2$? And what does $\mathbb{E}\left[ X^2 \right]$ mean? – Tomasz Bekas Feb 19 '17 at 22:23
  • 1
    Sorry I could have been more precise, I am taking $X_i$ to be identically distributed and then dropping the $i$ subscript, so that $\mathbb{E} \frac{1}{m} \sum_i x_i^2 = \frac{1}{m} \sum_i \mathbb{E} [X_i^2 ] = \frac{1}{m}\sum_i \mathbb{E} [ X^2 ] = \frac{1}{m} \cdot m \mathbb{E} [X^2]$ – Nadiels Feb 19 '17 at 22:27
  • 1
    equivalently I could have said $\mathbb{E}[X_i^2] = \mathbb{E}[X_1^2]$, for all $i$, but just calling it $E[X^2]$ saved writing more subscripts – Nadiels Feb 19 '17 at 22:28
  • Wow, this solution kind of blows my mind, thank you! How did you came up with this one $\left( \sum_i x_i \right)^2 = m \sum_i x_i^2 - \frac{1}{2}\sum_i \sum_j (x_i - x_j)^2$. I've never seen such an expansion of square of sum, I had to check it first to see if it's correct! – Tomasz Bekas Feb 19 '17 at 23:16
  • 1
    Hah the expansion blew my mind too and this looked like an opportunity to get to use it! I first found it in Kolgmogorov/Fomin Introductory Real Analysis where as an exercise you are asked to show the more general $\left( \sum_i a_i b_ i \right)^2 = \left( \sum_i a_i^2 \right) \left( \sum_j b_j^2 \right) - \frac{1}{2} \sum_{i} \sum_j (a_i b_j - a_j b_i )^2 $ – Nadiels Feb 19 '17 at 23:32
1

One way to get the equality is to evaluate the expected value of the unbiased estimator $s^2$. I use n instead of m and $ \overline X$ for the (unbiased) estimator of $\mu$.

$E(s^2)=E\left[\frac{1}{n-1}\sum_{i=1}^n (X_i-\overline X )^2\right]$

$=\frac{1}{n-1}E\left[\sum_{i=1}^n (X_i-\overline X)^2 \right] \quad | \pm \mu$

$=\frac{1}{n-1}E\left[\sum_{i=1}^n \left[(X_i-\mu)-(\overline X-\mu) \right]^2 \right] \quad$

multipliying out

$=\frac{1}{n-1}E\left[\sum_{i=1}^n \left[(X_i-\mu)^2-2(\overline X-\mu)(X_i-\mu)+(\overline X-\mu)^2 \right]\right] \quad$

writing for each summand a sigma sign

$=\frac{1}{n-1}E\left[\sum_{i=1}^n (X_i-\mu)^2-2(\overline X-\mu)\sum_{i=1}^n(X_i-\mu)+\sum_{i=1}^n(\overline X-\mu)^2 \right] \quad$

$=\frac{1}{n-1}E\left[\sum_{i=1}^n (X_i-\mu)^2-2(\overline X-\mu)\color{blue}{\sum_{i=1}^n(X_i-\mu)}+n(\overline X-\mu)^2 \right] \quad$

transforming the blue term

$\sum_{i=1}^n(X_i-\mu)=n\cdot \overline X-n\cdot \mu$

Thus $2(\overline X-\mu)\color{blue}{\sum_{i=1}^n(X_i-\mu)}=2(\overline X-\mu)\cdot (n\cdot \overline X-n\cdot \mu)=2n( \overline X- \mu)^2$

$=\frac{1}{n-1}E\left[\sum_{i=1}^n (X_i-\mu)^2-2n( \overline X- \mu)^2+n(\overline X-\mu)^2 \right] \quad$

$=\frac{1}{n-1}E\left[\sum_{i=1}^n (X_i-\mu)^2-n( \overline X- \mu)^2\right] \quad$

$=\frac{1}{n-1}\left[\sum_{i=1}^n E\left[(X_i-\mu)^2\right]-nE\left[( \overline X- \mu)^2\right]\right] \quad$

We know, that $E\left[(X_i-\mu)^2\right]=\sigma^2$ and $E\left[( \overline X- \mu)^2\right]=\sigma_{\overline x}^2=\frac{\sigma^2}{n}$ Thus we get

$=\frac{1}{n-1}\left[n \cdot \sigma ^2-n \frac{\sigma ^2}{n}\right]=\frac{1}{n-1} \sigma^2 \cdot (n-1)=\boxed{\sigma ^2=E(s^2)}$

If you would calculate $E(\hat \sigma)= E\left[\frac{1}{n}\sum_{i=1}^n (X_i-\overline X )^2\right]$ the result is almost the same but you have to multiply the result by n to neutralize $\frac{1}{n}$ and divide it by $\frac1{n-1}$

$E(\hat \sigma)=\frac{n}{n-1}\cdot \sigma^2$

callculus42
  • 30,550