12

Let $X_1,...,X_n$ be Bernoulli distributed with unknown parameter $p$.

My objective is to calculate the information contained in the first observation of the sample.

I know that the pdf of $X$ is given by $$f(x\mid p)=p^x(1-p)^{1-x}$$, and my book defines the Fisher information about $p$ as

$$I_X(p)=E_p\left[\left(\frac{d}{dp}\log\left(p^x(1-p)^{1-x}\right)\right)^2\right]$$

After some calculations, I arrive at

$$I_X(p)=E_p\left[\frac{x^2}{p^2}\right]-2E_p\left[\frac{x(1-x)}{p(1-p)}\right]+E_p\left[\frac{(1-x)^2}{(1-p)^2}\right]$$

I know that the Fisher information about $p$ of a Bernoulli RV is $\frac{1}{p(1-p)}$, but I don't know how to get rid of the X-values, since I'm calculating an expectation with respect to $p$, not $X$. Any clues?

StubbornAtom
  • 17,052

2 Answers2

10

\begin{equation} I_X(p)=E_p \left[\frac{X^2}{p^2}\right]-2E_p \left[ \frac{X - X^2}{p(1-p)} \right] + E_p \left[ \frac{X^2 - 2X + 1}{(1-p)^2}\right] \tag{1}. \end{equation} For a Bernoulli RV, we know \begin{align} E(X) &= 0(\Pr(X = 0)) + 1(\Pr(X = 1)) = p\\ E(X^2) &= 0^2(\Pr(X = 0)) + 1^2(\Pr(X = 1)) = p. \end{align} Now, replace in $(1)$, we get \begin{equation} I_X(p)=\frac{p}{p^2}-2\frac{0-0}{p(1-p)}+\frac{p-2p+1}{(1-p)^2} = \frac{1}{p}-\frac{p-1}{(1-p)^2} = \frac{1}{p} - \frac{1}{p-1} = \frac{1}{p(1 - p)}. \end{equation}

Sridhar Thiagarajan
  • 700
  • 2
  • 8
  • 23
Ahmad Bazzi
  • 12,076
  • thanks @MichaelHardy – Ahmad Bazzi Sep 16 '18 at 18:00
  • I always thought that when an expectation is written as $E_p$, for example, it means we treat the other variables (not p) as constants. Apparently not? –  Sep 16 '18 at 18:23
  • 1
    Yes i understand what you mean. I usually do not write $E_p()$. I write $E_{X \vert p}()$, where the expectation is taken with respect to the samples given the parameters you want to estimate, i.e. – Ahmad Bazzi Sep 16 '18 at 18:24
  • 1
    The expectation is taken with respect to any random variable (whether is $X$, $Y$ or whatever other r.v. $E_p$ only intends to remark the fact that your model for the distribution of $X$ is not fully specified, but it is uncertain up to a parameter $p$, and thus the corresponding expectations taken may depend on that value $p$. – Alejandro Nasif Salum Sep 16 '18 at 18:38
  • Also, if you put $I_X$, then it is the information of the random variable $X$ which has to be calculated, and so the formula should be in terms of $X$, not $x$. Otherwise, you're finding $I_x$. – Alejandro Nasif Salum Sep 16 '18 at 18:40
  • You seem to be mixing up $X$ and $x.$ In the expression $\Pr(X\le x)$ and elsewhere it is important to bear this distinction in mind. – Michael Hardy Sep 16 '18 at 19:19
  • 1
    Thanks @AlejandroNasifSalum for the edit – Ahmad Bazzi Sep 17 '18 at 23:28
4

Actually, the Fisher information of $X$ about $p$ is $$I_X(p)=E_p\left[\left(\frac{d}{dp}\log f(X\mid p) \right)^2 \right],$$ that is $$I_X(p)=E_p\left[\left(\frac{d}{dp}\log\left(p^X(1-p)^{1-X}\right)\right)^2\right].$$

I've only changed every $x$ by $X$, which may seem as a subtlety, but then you get $$I_X(p)=E_p\left(\frac{X^2}{p^2}\right)-2E_p\left(\frac{X(1-X)}{p(1-p)}\right)+E_p\left(\frac{(1-X)^2}{(1-p)^2}\right).$$

The expectation is there for the fact that $X$ is a random variable. So, for instance: $$E_p\left(\frac{X^2}{p^2}\right)=\frac{E_p\left(X^2\right)}{p^2}=\frac{p}{p^2}=\frac1p.$$

Here I used the fact that $E_p(X^2)=p$, which can easily be seen as $$E_p(X^2)=0^2\cdot p_X(0)+1^2\cdot p_X(1)=0^2(1-p)+1^2p=p,$$ or by the observation that $X\sim \operatorname{Be}(p) \implies X^n\sim \operatorname{Be}(p)$ as well. Then you can go on with the remaining terms.

Additionally, an equivalent formula can be proved for $I_X(p)$ given the second derivative of $\log f$ is well defined. This is $$I_X(p)=-E_p\left(\frac{d^2}{dp^2}\log f(X\mid p) \right),$$ and many times you'll get simpler expressions. In this case, for instance, you get $$I_X(p)=-E_p\left(\frac{d^2}{dp^2}\log p^X(1-p)^{1-X}\right)=$$ $$=-E_p\left(-\frac X{p^2}-\frac{1-X}{(1-p)^2} \right) = \frac {E_p(X)}{p^2}+\frac{E_p(1-X)}{(1-p)^2}=$$ $$=\frac {p}{p^2}+\frac{1-p}{(1-p)^2}=\frac 1p+\frac 1{1-p}=\frac 1{p(1-p)},$$ as desired.

Michael Hardy
  • 1
Alejandro Nasif Salum
  • 5,703
  • 1
    agreed about the simpler expression with the negative expectation of 2nd derivative of log-likelihood of 1 observation. In the course I'm following, this is how Fisher Information is defined. Makes life simpler indeed :) – alisa Jan 23 '21 at 06:30
  • Yes, I give my students both formulas so they can choose. In cases in which the derivatives get too complicated, the first one might be a better choice, but in most usual examples that is not the case. – Alejandro Nasif Salum Jan 24 '21 at 18:26