Proof variance of Geometric Distribution

Question

I have a Geometric Distribution, where the stochastic variable $X$ represents the number of failures before the first success.

The distribution function is $P(X=x) = q^x p$ for $x=0,1,2,\ldots$ and $q = 1-p$.

Now, I know the definition of the expected value is: $E[X] = \sum_{i}{x_i p_i}$

So, I proved the expected value of the Geometric Distribution like this:

$E[X]=\sum _{ i=0 }^{ \infty }{ iP(X=i) } = \sum _{i=0}^{\infty}{i q^i p} = p\sum _{i=0}^{\infty}{i q^i} = pq \sum _{i=0}^{\infty}{iq^{i-1}}$

$\qquad = pq \sum _{i=0}^{\infty}{\frac{d}{dq}q^i} = pq \frac{d}{dq}(\sum _{i=0}^{\infty}{q^i}) = pq \frac{d}{dq}(\frac{1}{1-q})$

$\qquad = pq \frac{1}{(1-q)^2} = \frac{pq}{p^2} = \frac{q}{p}$

So now, I would like to prove that $Var[X] = \frac{q}{p^2}$. I know I have to use a simular trick as above (with the derivation).

$Var[X] = E[X^2] - E[X]^2 = \sum _{i=0}^{\infty}{i^2 q^i p} - (\frac{q}{p})^2 = p \sum _{i=0}^{\infty}{i^2 q^i} - (\frac{q}{p})^2 = pq \sum _{i=0}^{\infty}{i^2 q^{i-1}} - (\frac{q}{p})^2$

$\qquad = pq \sum _{i=0}^{\infty}{\frac{d}{dq}i q^i} - (\frac{q}{p})^2 = pq \frac{d}{dq} \sum _{i=0}^{\infty}{iq^i}-(\frac{q}{p})^2$

Then I'm stuck. How can I get another $q$ out of the sum? Won't it mess up the first derivation?

Answer 1

I have a proof which follows the approach of @Math1000 but it in a slightly different way. It may be useful if you're not familiar with generating functions.

However, I'm using the other variant of geometric distribution. In my case $X$ is the number of trials until success. Therefore $E[X]=\frac{1}{p}$ in this case. Anyways both variants have the same variance.

So assuming we already know that $E[X]=\frac{1}{p}$. Then the variance can be calculated as follows: $$ Var[X]=E[X^2]-(E[X])^2=\boxed{E[X(X-1)]} + E[X] -(E[X])^2 = \boxed{E[X(X-1)]} + \frac{1}{p} - \frac{1}{p^2} $$ So the trick is splitting up $E[X^2]$ into $E[X(X-1)]+E[X]$, which is easier to determine. To determine $\boxed{E[X(X-1)]}$ we have to determine the value of the following series for $p\in(0,1)$: $$ \sum_{k=1}^\infty k(k-1)p(1-p)^{k-1} $$

Here's how it can be done (as an alternative to Math1000's approach): $$ \begin{align} \sum_{k=1}^\infty k(k-1)p(1-p)^{k-1} &= p\sum_{k=1}^\infty k(k-1)(1-p)^{k-1} \qquad\text{Subst. }q:=(1-p)\\\\ &= p\sum_{k=1}^\infty (k-1)kq^{k-1} \\\\ &= p\frac{d}{dq}\left(\sum_{k=1}^\infty (k-1)q^k\right) \\\\ &= p\frac{d}{dq}\left(q^2\sum_{k=1}^\infty (k-1)q^{k-2}\right) \\\\ &= p\frac{d}{dq}\left(q^2\sum_{k=2}^\infty (k-1)q^{k-2}\right) \\\\ &= p\frac{d}{dq}\left(q^2\frac{d}{dq}\left(\sum_{k=2}^\infty q^{k-1}\right)\right) \\\\ &= p\frac{d}{dq}\left(q^2\frac{d}{dq}\left(\sum_{k=1}^\infty q^{k}\right)\right) \\\\ &= p\frac{d}{dq}\left(q^2\frac{d}{dq}\left(\frac{1}{1-q}-1\right)\right) \\\\ &= p\frac{d}{dq}\left(\frac{q^2}{(1-q)^2}\right) \\\\ &= p\left(\frac{-2q}{(q-1)^3}\right)\qquad\text{Backsub. }q=(1-p) \\\\ &= p\left(\frac{-2(1-p)}{((1-p)-1)^3}\right) = p\left(\frac{-2+2p}{-p^3}\right) \\\\ &= \frac{-2+2p}{-p^2} =\frac{2(p-1)}{-p^2} = \frac{2(1-p)}{p^2}. \\\\ \end{align} $$ Now putting the result back into the equation for $Var[X]$ gives us: $$ Var[X]=\boxed{E[X(X-1)]} + E[X] -(E[X])^2 =\frac{2(1-p)}{p^2} + \frac{1}{p} - \frac{1}{p^2} = \frac{2-2p+p-1}{p^2} = \frac{1-p}{p^2}. $$

Answer 2

No answer to your question but a suggestion to follow an alternative route (too much for a comment).

Let $S$ denote the event that the first experiment is a succes and let $F$ denote the event that the first experiment is a failure. Then make use of: $$\mathbb EX^n=\mathbb E(X^n|S)P(S)+\mathbb E(X^n|F)P(F)=\mathbb E(1+X)^nq$$ This for $n=1$ and $n=2$ respectivily.

It leads to expressions for $\mathbb EX$, $\mathbb EX^2$ and consequently $\text{Var}X=\mathbb EX^2-(\mathbb EX)^2$.

Answer 3

$\mathbb E[X] = \frac{1-p}p$ as you computed above. Here is a trick to make the computation of $\mathrm{Var}(X)$ easier: $$ \mathrm{Var}(X) = \mathbb E[X^2] - \mathbb E[X]^2 = \mathbb E[X(X-1)] + \mathbb E[X] - \mathbb E[X]^2. $$ Since the generating function of $X$ is $$\begin{align*} P(s) &:= \mathbb E\left[s^X\right]\\ &= \sum_{n=0}^\infty (1-p)^n p s^n\\ &= p\sum_{n=0}^\infty ((1-p)s)^n\\ &= \frac p{1-(1-p)s}, \end{align*}$$ we have $$\begin{align*} \mathbb E[X(X-1)] &= \lim_{s\uparrow1} P''(s)\\ &= \lim_{s\uparrow1} \frac{2p(1-p)^2}{\left(1-(1-p)s\right)^3}\\ &= \frac{2p(1-p)^2}{p^3}\\ &= \frac{2(1-p)^2}{p^2} \end{align*}$$ Hence $$\begin{align*} \mathrm{Var}(X) &= \frac{2(1-p)^2}{p^2} +\frac{1-p}p - \left(\frac{1-p}p\right)^2\\ &= \frac{2(1-p)^2 + p(1-p) -(1-p)^2 }{p^2}\\ &= \frac{(1-p)(1-p+p)}{p^2}\\ &= \frac{1-p}{p^2}. \end{align*}$$

Answer 4

I just happened to see it later, but actually you were really close. You just have to use the derivation-trick another time. So continuing from where you've been you'd do: $$ \begin{align*} \ldots &= pq \frac{d}{dq}\left[ \sum _{i=0}^{\infty}{iq^i}\right]-(\frac{q}{p})^2 \\ &= pq \frac{d}{dq}\left[ \sum _{i=0}^{\infty}q{iq^{i-1}}\right]-(\frac{q}{p})^2 \\ &= pq \frac{d}{dq}\left[ q\sum _{i=0}^{\infty}\frac{d}{dq}\left[q^{i}\right]\right]-(\frac{q}{p})^2 \\ &= pq \frac{d}{dq}\left[ q\frac{d}{dq}\left[\sum _{i=0}^{\infty}{q^{i}}\right]\right]-(\frac{q}{p})^2 \\ &=\ldots \end{align*} $$ Then you just continue as follows:

1. Use the rule for the geometric series
2. Derivate with respect to $q$
3. Derivate another time with respect to $q$
4. Then you just have to collect the terms and you should get there.

Answer 5

It's very similar to the proof of $\mathbb E[X] = \frac{q}{p}$ that you've already worked out.

Given $X \sim \mathcal{Geo}(k;\ p)\ ,where\ k \in \{0, 1, 2, 3, ..., K\} and\ p \in (0,1]$, below is the the proof of $Var[X]$:

$$\begin{align} Var[X] &= E[X^2]-E[X]^2 \\ \text{linearity of expectation:}\qquad &= E[X(X-1)] + E[X] - E[X]^2 \\ \text{law of the unconscious statistician:}\qquad &= \sum_{k=0}^\infty k(k-1) \ \mathcal{Geo}(k;\ p)+E[X]-E[X]^2 \\ \text{If we let } \gamma =E[X]-E[X]^2 \text{ and }q=1-p:\qquad &=\sum_{k=0}^\infty\ k(k-1)\ pq^{k}+\gamma \\ &=pq^2\sum_{k=0}^\infty\ k(k-1)q^{k-2}+\gamma \\ &=pq^2\sum_{k=0}^\infty\ \frac{\partial^2}{\partial q^2}(\int_0^1k(k-1)q^{k-2}\ dq^2)+\gamma \\ \text{power rule of second order derivative:}\qquad &=pq^2\sum_{k=0}^\infty\ \frac{\partial^2}{\partial q^2}q^k+\gamma \\ \text{linearity of differentiation:}\qquad &=pq^2\frac{\partial^2}{\partial q^2}\sum_{k=0}^\infty\ q^k+\gamma \\ \text{recall the sum of geometric series}\sum_{k=0}^\infty\ q^k=\frac{1-q^{k+1}}{1-q}:\qquad &=pq^2\frac{\partial^2}{\partial q^2}\frac{1-q^{k+1}}{1-q}+\gamma \\ &=pq^2\frac{\partial^2}{\partial q^2}\frac{1}{1-q}+\gamma \\ &=pq^2\frac{2}{(1-q)^3}+\gamma \\ &=q^2\frac{2}{p^2}+\frac{q}{p}-\frac{q^2}{p^2} \\ &=\frac{2q^2+pq-q^2}{p^2} \\ &=\frac{q(q+p)}{p^2} \\ &=\frac{1-p}{p^2} \end{align}$$ proved.

Answer 6

I'm using the variant of geometric distribution the same as @ndrizza. Therefore $E[X]=\frac{1}{p}$ in this case. $$E[X^2]=E[X^2|X^2=1]P(X^2=1)+E[X^2|X^2>1]P(X^2>1)$$ notice that $E[X^2|X^2=1]=1$, and the condition $X^2>1$ is the same as $X>1$, we got: $$E[X^2]=p+E[X^2|X>1]P(X>1)$$ $$E[X^2]=p+(1-p)E[X^2|X>1]$$ Consider that $$E[X^2|X>1]=E[((X-1)^2+2(X-1)+1)|X-1>0]$$ $$E[X^2|X>1]=E[(X-1)^2|X-1>0]+2E[(X-1)|X-1>0]+1$$ $$E[X^2|X>1]=E[X^2]+2E[X]+1$$ $$E[X^2|X>1]=E[X^2]+\frac{2}{p}+1$$ Put this back to $E[X^2]=p+(1-p)E[X^2|X>1]$, we got: $$E[X^2]=p+(1-p)(E[X^2]+\frac{2}{p}+1)$$ $$E[X^2]=\frac{2-p}{p^2}$$ Put this to $Var[X]=E[X^2]-E[X]^2$, we got $$Var[X]=\frac{1-p}{p^2}$$

Answer 7

It's my first post here, so please forgive the mess.

You can use moment generating function to find the first two moments of the distribution, i.e. $E[x]$ and $E[x^2]$.

First, let's see what the mgf for the geometric distribution looks like. By the definition: $$M(t) = \sum_{x=0} e^{tx}p(x)$$ Therefore, $$M(t) = \sum_{x=0} e^{tx}q^xp = p\sum_{x=0}(qe^t)^x$$ Now, assuming $qe^t < 1$, the infinite sum simplifies to $\frac{1}{1-qe^t}$ and we get: $$M(t) = \frac{p}{1-qe^t}$$

By definition, $$E[X^r] = M^{(r)}(0)$$ First, $$M'(t) = -\frac{p}{(1-(1-p)e^t)^2}(-(1-p))e^t = \frac{p(1-p)e^t}{(1-(1-p)e^t)^2}$$ And, $$E[X] = M'(0) = \frac{p(1-p)}{p^2} = \frac{1-p}{p}$$ Now, let's calculate the second derivative of the mgf w.r.t $t$: $$M''(t) = \frac{p(p-1)\left[(p-1)e^t-1\right]e^t}{((p-1)e^t+1)^3}$$ and $$E[X^2] = M''(0) = \frac{(p-2)(p-1)}{p^2}$$ And finally: $$Var[X] = E[X^2] - (E[X])^2 = \frac{(1-p)\left[(2-p) - (1-p)\right]}{p^2} = \frac{1-p}{p^2}$$