Find the bias for the Maximum-likelihood estimator

Question

Let $X_1,...,X_n$ be a random sample from the pdf $$f(x|\theta) = \theta x^{\theta-1} , 0 \leq x \leq 1, \theta >0.$$

I found the Maximum-likelihood estimator of $\theta$ is $$\hat{\theta} = \frac{-n}{\sum_{i=1}^N \ln(X_i)}.$$ Can anyone confirm that this is right?

Then, I want to determine whether $\hat{\theta}$ has bias. My approach is to calculate ${\bf E}[\hat{\theta}] = {\bf E}\left[\frac{-n}{\sum_{i=1}^N \ln(X_i)}\right]$...Then I am stuck. Could someone help me with this?

score 4 · Accepted Answer · edited Apr 17 '21 at 00:44

4

You got the right answer for the MLE. To find the expectation of $\hat{\theta}$, it will help to first find the distribution of the transformation $Y=-\log(X_i)$. It's a well known distribution, and so will be $-\sum \log(X_i)$ as well as $-1/\sum \log(X_i)$.

You have already found that the distribution of $Y_i=-\log(X_i)$ is exponential($1/\theta$).

According to section 4.2 of Pitman, "Probability", the sum of n iid exponential($1/\theta$) distributions is gamma(n, $1/\theta$). Thus, Z =$\sum Y_i=-\sum \log(X_i)$ is distributed gamma(n,$1/\theta$) where the pdf of Z is

$\dfrac{\theta^n}{\Gamma(n)}z^{n-1}e^{-z\theta}$ with mean $E(Z)=n/\theta$.

Notice that since a pdf must integrate to 1, it is straightforward to show

$\int_0^{\infty}z^{n-1}e^{-z\theta}dz=\dfrac{\Gamma(n)}{\theta^n}$. Call this result 1.

This will be useful in the following step to find $E(1/Z)$.

$E(1/Z) = \int_0^{\infty}1/z\dfrac{\theta^n}{\Gamma(n)}z^{n-1}e^{-z\theta}dz$

$= \dfrac{\theta^n}{\Gamma(n)} \int_0^{\infty}z^{(n-1)-1}e^{-z\theta}$

$= \dfrac{\theta^n}{\Gamma(n)}\dfrac{\Gamma(n-1)}{\theta^{n-1}}$ by result 1.

$=\dfrac{\theta\Gamma(n-1)}{\Gamma(n)}$

Now, according to Casella and Berger, Statistical Inference pg 99, a useful property of the gamma function is that

$\Gamma(\alpha+1)=\alpha\Gamma(\alpha)$, thus

$\Gamma(n)=\Gamma((n-1)+1) = (n-1)\Gamma(n-1)$ and

$E(1/Z) = \dfrac{\theta\Gamma(n-1)}{\Gamma(n)} =\dfrac{\theta\Gamma(n-1)}{(n-1)\Gamma(n-1)}= \theta/(n-1) $

It's now straightforward to find $E(n/Z) = n*E(1/Z) = n\theta/(n-1) $.

edited Apr 17 '21 at 00:44

The Pointer

4,182

answered May 13 '14 at 06:55

jsk

543

I found that $log(X_i)$ has an exponential distribution with mean $1/\theta$. How does this help? – afsdf dfsaf May 13 '14 at 14:51
Do you know what the distribution is of a sum of exponential distributions? – jsk May 13 '14 at 14:54
is it going to be gamma distribution? – afsdf dfsaf May 13 '14 at 14:58
I have also showed that $2\theta \sum_{i=1}^n log(X_i)$ has a $\chi^2$ distribution with $2n$. degrees of freedom. Will this help? – afsdf dfsaf May 13 '14 at 15:06
Yes. And do you know what happens when you multiply gamma by a constant $1/n$? Or when you take the inverse of a random variable with a gamma distribution? – jsk May 13 '14 at 15:07
not sure....could you help on this? – afsdf dfsaf May 13 '14 at 15:09
I could help, but I think I have given enough direction to finish the problem. No need to make it into a $\chi ^2$ – jsk May 13 '14 at 15:22
When you say "multiply gamma by a constant 1/n, why is 1/n since n is in the numerator? ...Also, is $$\frac{-1}{\sum_{i=1}^n ln(X_i)}$$ or just $\sum_{i=1}^n ln(X_i)$a gamma distribution? – afsdf dfsaf May 13 '14 at 15:26
Right, you could invert it then multiply by n, or multiply by 1/n then invert – jsk May 13 '14 at 15:31
The inverse of a gamma is also a standard result. – jsk May 13 '14 at 15:34
will still be a gamma distribution. isn't it? – afsdf dfsaf May 13 '14 at 15:50
No. The inverse of a gamma is not a gamma, though the name of the distribution does have the word gamma in it – jsk May 13 '14 at 15:55
is it something I have to show with a lot of math or it is just some well-known fact? – afsdf dfsaf May 13 '14 at 16:50
There's barely any math necessary to perform the transformation and derive the distribution. It's called an inverse gamma, or an inverted gamma. you can find the mean and variance on wikipedia. I hope I have provided you with sufficient assistance. Please remember to upvote and accept if I was helpful! – jsk May 13 '14 at 16:57
so the mean for $\sum_{i=1}^n ln(x_i)$ is n/$\theta$? – afsdf dfsaf May 13 '14 at 17:12
Correct. But be careful because the mean of the inverse will not be one over the mean – jsk May 13 '14 at 17:50
you mean $\frac{-n}{\sum_{i=1} ln(x_i)}$ will not be $\frac{-n}{n/\theta}$? – afsdf dfsaf May 13 '14 at 18:16
Correct. In general, as a result of jensen's inequality, $E(1/X) \neq 1/E(X)$ – jsk May 13 '14 at 18:25
so, how can I find $ E[\frac{-n}{\sum_{i=1}^n ln(x_i)}]$? – afsdf dfsaf May 13 '14 at 18:27
Let $Y = -n/\sum ln(x_i)$. Then derive the distribution of $Y$ to be an inverse gamma. Then use standard results about the mean of an inverse gamma. – jsk May 13 '14 at 18:34
If we derive the distribution of Y, is it going to be $$E[\frac{-n}{\sum_{i=1}^n ln(X_i)}] = -n/x [ \frac{(1/\theta)^n}{\Gamma(n)} x^{-n-1} e^{\frac{-1}{\theta x}}]$$? – afsdf dfsaf May 13 '14 at 18:44
No, that doesn't look quite right. See here – jsk May 13 '14 at 18:54
This is what I follow ... i first take out -n, then multiply by 1/x since we are dealing with expected value. Then, $\alpha= n$, and $\beta = 1/\theta$...so the inside the [...] is just the pdf of invert of gamma distribution. what's wrong? – afsdf dfsaf May 13 '14 at 18:58
Sorry, but I do not follow what you are saying and will not be available for the next 6 hours. I hope that you are able to work through the details with the help of the link I sent you. Please let me know if you still end up requiring assistance. – jsk May 13 '14 at 19:09
I am still struck. Could you let me know which part of my answer you do you follow? – afsdf dfsaf May 14 '14 at 01:36
It looks like you were trying to find the expectation instead of deriving the distribution of Y. You can do this problem either way. – jsk May 14 '14 at 01:50
is my setup for finding expectation correct then? Which way is easier? – afsdf dfsaf May 14 '14 at 01:51
If you already know that if $X \sim $ gamma implies $1/X \sim$ Inverse Gamma, then the problem is basically done because Inverse Gamma is a well-known distribution and you can look up the mean on the link I posted. – jsk May 14 '14 at 02:31
$\alpha = n$ and $\beta = 1/\theta$, right? – afsdf dfsaf May 14 '14 at 02:37
Careful, you're getting close though. What were the parameters for the distribution of $-\sum log(X_i)$? – jsk May 14 '14 at 02:41
$\alpha$ and $\beta$ since this is gamma distribution? – afsdf dfsaf May 14 '14 at 02:42
Yes, $alpha=n$ and $\beta = 1/\theta$?, right? So what is the distribution of $- \sum log(X_i) / n$ – jsk May 14 '14 at 02:45
I am thinking if $E[\hat{\theta}]$ not equal to $\theta$. Then, there is bias. If they are equal, no bias exists. Do we still need to care the distribution (or I do not know what you are referring to)? – afsdf dfsaf May 14 '14 at 02:47
By the way, what we want to show is $E[-n/\sum log(X_i)]$ right? – afsdf dfsaf May 14 '14 at 02:48
You are correct that it is biased. And so you know that $-1/\sum log(X_i)$ is inverse gamma with $\alpha=n$ and $\beta=1/\theta$, right? – jsk May 14 '14 at 02:50
So what is $E( - 1/\sum log(X_i))$? – jsk May 14 '14 at 02:50
$\frac{1/\theta}{n-1}$? – afsdf dfsaf May 14 '14 at 02:54
Hmm, close use $\alpha=n$ and $\beta=\theta$. – jsk May 14 '14 at 02:56
is it because we use inverse of gamma distribution...so we use $\beta = \theta$ instead of $1/\theta$? – afsdf dfsaf May 14 '14 at 02:57
No, the parameterization just gets confusing because there are two definitions of gamma which results in different expressions for the mean. – jsk May 14 '14 at 02:59
wait a minute: $log(X_i)$ has an exponential distribution with mean $1/θ$. Can you explain why $\beta = \theta$ instead of $1/\theta$? – afsdf dfsaf May 14 '14 at 03:02
Depending on the parameterization the mean of a gamma is either $\alpha/\beta$ or $\alpha*\beta$ – jsk May 14 '14 at 03:06
is it possible to not use the property of mean for the inverse of gamma distribution and just use gamma distribution mean property? – afsdf dfsaf May 14 '14 at 03:08
Yes, you should get $\theta/(n-1)$ for $E( -1/ \sum log(X_i))$, and thus $n*\theta/(n-1)$ for $E( -n / \sum log(X_i))$. – jsk May 14 '14 at 03:10
$E[-n/\sum log (X_i)] = \frac{-n}{n-1} \theta$?? – afsdf dfsaf May 14 '14 at 03:12
No, $n\theta / (n-1)$. Careful of the negative sign. – jsk May 14 '14 at 04:10
why it disappears? – afsdf dfsaf May 14 '14 at 04:12
It doesn't! $-\sum log(X_i)$ has a gamma distribution. – jsk May 14 '14 at 04:13
The negative sign is part of the transformation! – jsk May 14 '14 at 04:14
that's true...to sum up...when we focus on inverse gamma distribution, $\alpha = n$ and $\beta = \theta$...so we use the property of mean $\frac{\beta}{\alpha-1}$...which gives us so $E[-1/\sum log (X_i)] = \theta/(n-1)$? And therefore, $E[-n/\sum log (X_i)] = \frac{n}{n-1} \theta$? But for the use of gamma distribution only, we set $\alpha = n$ and $\beta = 1/\theta$, right?...if so, we use the property of mean $\beta\alpha$...which gives us so $E[-\sum log (X_i)] = 1/\theta (n)$? And therefore, $E[-n/\sum log (X_i)] = \theta$?...Why the two approaches give different results? – afsdf dfsaf May 14 '14 at 04:20
This completely depends on your definition of gamma! Look at the two definitions here. Look at the two definitions of the pdf. – jsk May 14 '14 at 04:21
Based on my answer above, which one is correct, then? – afsdf dfsaf May 14 '14 at 04:23
1

whoa, wait a sec. You have to pay attention to the details. $E(1/X)$ DOES NOT EQUAL $1/E(X)$!!!! – jsk May 14 '14 at 04:23
that means I cannot use the gamma distribution only approach right> – afsdf dfsaf May 14 '14 at 04:24
1

YOU CAN USE THE GAMMA APPROACH, but you CANNOT claim that $ E(-1/\sum log(X_i)) = 1/E(-\sum log(X_i))$ THIS IS NOT TRUE BECAUSE OF JENSEN'S INEQUALITY – jsk May 14 '14 at 04:28
so if I want to use gamma approach, what should I do? – afsdf dfsaf May 14 '14 at 04:33
Learn how to take expectations by pulling out the appropriate constant and formulate the kernel of the appropriate gamma distribution when you integerate. – jsk May 14 '14 at 04:34
you lost me here! but let nail down the bias here: the bias will be (n-1)/n, right? – afsdf dfsaf May 14 '14 at 04:38
bias is generally defined by $E(\hat{\theta}-\theta ) = E(\hat{\theta})- \theta = n\theta/(n-1)- \theta $. I leave the algebra for you. – jsk May 14 '14 at 04:51
$\theta/(n+1) $ is what you got? – afsdf dfsaf May 14 '14 at 04:56
$\frac{n\theta}{n-1} - \theta = \frac{ n\theta - (n-1)\theta}{n-1} = \theta/(n-1)$ – jsk May 14 '14 at 05:00
I'm very sorry you are having so much trouble with this work. I sincerely hope that you are able to pick up the necessary skills you need and pass your class. – jsk May 14 '14 at 05:02
Sure I will do it. I have several details I still have questions. Let go back to the "Learn how to take expectations by pulling ....when you integrate" ... can you elaborate on it? – afsdf dfsaf May 14 '14 at 05:02
also, I am still a little confused by why $\alpha = n$ – afsdf dfsaf May 14 '14 at 05:06
Meanwhile, I have a similar question listed below. If you would like to help out with this, I will appreciate it. http://math.stackexchange.com/questions/792554/find-the-pdf-of-prod-i-1n-x-i-where-x-is-are-independent-uniform-0-1 – afsdf dfsaf May 14 '14 at 05:17
Since $-log(X_i) \sim exp(1/\theta)$ so that $E(-log(X_i)) = 1/\theta$, you get $-\sum log(X_i) \sim gamma( n, 1/\theta)$ where $E(-\sum log(X_i)) = n/\theta$ – jsk May 14 '14 at 05:18
With this result I just gave you for the sum of iid exponentials, you should be able to do your other problem as well! – jsk May 14 '14 at 05:22
For the question i have on other post, after $\sum -ln(X_i)$ it should be a Gamma(n,1), right? – afsdf dfsaf May 14 '14 at 05:29
Yes, that is correct. – jsk May 14 '14 at 05:33
how to convert that back to the $\prod X_i$ uniform independent random variables – afsdf dfsaf May 14 '14 at 05:42
transformation. Define $Y = \sum log(X_i) = log( \prod X_i)$. Then perform the transformation $e^Y$ – jsk May 14 '14 at 05:49
Recall the rule that $e^{log(X)} = X$ – jsk May 14 '14 at 05:50
how does this relates to the pdf for Y which is $\int_0^{\infty} \frac{1}{\Gamma n} y^{n-1} e^{-y} dy$? – afsdf dfsaf May 14 '14 at 05:52
should I $$e^{this pdf}$$? – afsdf dfsaf May 14 '14 at 05:54
Define $Z=e^Y$ and find the distribution of $Z$. Notice that the distribution of $Z$ is the distribution of $\prod X_i$. – jsk May 14 '14 at 05:54
i know. this is my question how can I get my pdf for Y to find Z? or I do not need pdf for Y? – afsdf dfsaf May 14 '14 at 05:56
Surely your teacher went over how to find the pdf for a transformation? – jsk May 14 '14 at 05:57
I know . it is weird to exponentiate a pdf for Y to Z? or this is a wrong approach? – afsdf dfsaf May 14 '14 at 05:59
You exponentiate a random variable. You don't exponentiate the pdf. – jsk May 14 '14 at 06:00
You should post this as a separate question. Say you have the pdf for Y which is gamma and you want to find the pdf for $e^Y$ – jsk May 14 '14 at 06:00
the pdf for Z is $$\int_1^\infty \frac{1}{\Gamma (n)} ln(z)^{n-1} \frac{1}{z^2} dz$$? – afsdf dfsaf May 14 '14 at 06:06
I create a new post link listed below:http://math.stackexchange.com/questions/794181/the-pdf-for-y-which-is-gamma-and-how-to-find-the-pdf-for-z-ey – afsdf dfsaf May 14 '14 at 06:16

Find the bias for the Maximum-likelihood estimator

1 Answers1