Proof that $\frac{(\bar X-\mu)}{\sigma}$ and $\sum_{i=1}^n\frac{(X_i-\bar X)^2}{\sigma^2}$ are independent

Question

Let $X_i\sim N(\mu,\sigma^2)$ ; where$[i=1,2,\ldots,n]$

$Z_i\sim N(0,1)$ ; where$[i=1,2,\ldots,n]$

Proof that $\bar Z=\frac{(\bar X-\mu)}{\sigma}$ and $\sum_{i=1}^{n}(Z_i-\bar Z)^2=\sum_{i=1}^n\frac{(X_i-\bar X)^2}{\sigma^2}$ are independent, which implies $\bar X$ and $\sum_{i=1}^n(X_i-\bar X)^2$ are independent.

MY ATTEMPT:

I considered $n=2$, and

$$M_{X_1+X_2}(t_1)=M_{X_1}(t_1)M_{X_2}(t_1)$$

$\ast$I did so for the proof but does the statement $X_i\sim N(\mu,\sigma^2)$ ; where$[i=1,2,\ldots,n]$ imply that $X_i's$ are independent?

Moment Generating Function of $X_1+X_2$ and $X_1-X_2$ are $$M_{X_1+X_2}(t_1)=e^{2\mu t_1+\sigma^2 t_1^2}$$ $$M_{X_1-X_2}(t_2)=e^{\sigma^2 t_2^2}$$ respectively.

Also,

$$M_{X_1+X_2,X_1-X_2}(t_1,t_2)=e^{2\mu t_1+\sigma^2 t_1^2}e^{\sigma^2 t_2^2}=M_{X_1+X_2}(t_1)M_{X_1-X_2}(t_2)$$

since the joint moment generating function factors into the product of the marginal moment generating functions, so $X_1+X_2$ and $X_1-X_2$ are independent which implies that:

$\bullet$ Since $\bar Z=\frac{(\bar X-\mu)}{\sigma}$ is only a function of $X_1+X_2$ and $\sum_{i=1}^{2}(Z_i-\bar Z)^2=\sum_{i=1}^2\frac{(X_i-\bar X)^2}{\sigma^2}$ is only a function of $X_1-X_2$ ,so

$\bar Z=\frac{(\bar X-\mu)}{\sigma}$ and $\sum_{i=1}^{n}(Z_i-\bar Z)^2=\sum_{i=1}^n\frac{(X_i-\bar X)^2}{\sigma^2}$ are independent.

$\bullet$ Since $\bar X$ is only a function of $X_1+X_2$ and $S^2=\frac{1}{2-1}\sum_{i=1}^2(X_i-\bar X)^2$ is only a function of $X_1-X_2$ ,so

Sample mean,$\bar X$ and sample variance,$S^2$ are independent.

$\diamond\diamond\diamond$ We accept the above independence for any arbitrary $n$.

Is the procedure correct?

Answer 1

The way to do this is to show that $\bar X=(X_1+\cdots+X_n)/n$ is independent of $$ (X_1-\bar X, \ldots, X_n-\bar X). $$ Notice that the orthogonal projection of $(X_1,\ldots,X_n)$ onto the $1$-dimensional space $\{ (x,x,x,\ldots,x) : a \in \mathbb R\}$ is $(\bar X,\ldots, \bar X)$, and the orthogonal projection onto the $(n-1)$-dimensional space $x_1+\cdots+x_n=0$ is $(X_1-\bar X, \ldots, X_n-\bar X)$, and those two spaces are orthogonal to each other. Notice also that the probability distribution of $(X_1,\ldots,X_n)$ is spherically symmetric about the point $(\mu,\mu,\ldots,\mu)$, which is in the $1$-dimensional space mentioned above, and which gets mapped to $(0,0,0,\ldots,0)$ by the second projection.

Another way to look at it is this: The random vector $(X_1,\ldots,X_n)$ is jointly normally distributed, i.e. it is so distributed that every linear combination $a_1X_1+\cdots+a_nX_n$ has a $1$-dimensional normal distribution. That implies that any two linear combinations of $X_1,\ldots,X_n$ that are uncorrelated are independent. So then notice that $$ \operatorname{cov}\left( \bar X, \begin{bmatrix} X_1 - \bar X \\ \vdots \\ X_n - \bar X\end{bmatrix}\right) = \begin{bmatrix} 0 \\ \vdots \\ 0 \end{bmatrix}. $$

Answer 2

One could use Cochran's theorem to prove this. Let $J$ be the $n\times n$ matrix of all ones. WLOG, $X\sim N(0,I_n)$. If $Y\sim N(\mu,\sigma^2 I_n)$, then we can standardize to get $(Y-\mu)/\sigma\sim N(0,I_n)$.

$$\sum_{i=1}^n X_i^2 = \sum_{i=1}^n (X_i - \bar{X})^2 + \frac{ (\sum_{i=1}^n X_i)^2 }{n} \\ \Rightarrow X'I_n X = X'(I_n - \frac{1}{n}J)X + X'(\frac{1}{n}J)X.$$

$I_n - \frac{1}{n}J$ has rank $n-1$, $\frac{1}{n}J$ has rank $1$, and $I_n$ has rank $n$, and they’re all PSD, thus the required conditions for Cochran's are satisfied. Thus $\sum_{i=1}^n (X_i - \bar{X})^2$ and $\frac{ (\sum_{i=1}^n X_i)^2 }{n}$ are independent and immediately, so are $\frac{\sum_{i=1}^n (X_i - \bar{X})^2}{n-1}$ and $\frac{\sum_{i=1}^n X_i}{n}=\bar{X}$ and we're done.

The last statement is because, for functions $f$ and $g$, $f(X)$ and $g(Y)$ are independent if $X$ and $Y$ are independent.

Also note that $\sum_{i=1}^n (X_i - \bar{X})^2\sim \chi^2(n-1)$ and $\frac{ (\sum_{i=1}^n X_i)^2 }{n}\sim \chi^2(1)$ from Cochran's as well.

================================================================== EDIT

Coming to think about this more, I have my doubts that we can use the above because I can't think of a univariate function that sends $\frac{ (\sum X_i) ^2 }{ n}$ to $\sum X_i /n$. $$\sqrt{ \frac{ (\sum X_i) ^2}{n}} * \frac{1}{\sqrt{n}} = \frac{\left| \sum X_i \right|}{n} \neq \frac{\sum X_i}{n} = \bar{X}.$$ However, I see the same reasoning as that of my answer in the literature. I just don't think it's correct.

For example, you could have a case where $Y$ and $X^2$ were independent but $Y$ and $X$ are not.

Let $Y=2X$ and $X=\pm 1$ with equal probabilities. Then $X^2=1$ a.s. and independent of $Y$. However, $Y$ depends on $X$.

So if someone can tell me why the Cochran's method would still work, that'd be helpful.

Instead, I'll prove the independence of sample mean and variance in a different way. I use a theorem about linear and quadratic forms that says $AX$ and $X'BX$ are independent if $AB=0$, assuming $X\sim N(0,I_n)$ and $B$ is symmetric and idempotent. Here, let $A=(1/n,\ldots,1/n)^{\top}$ and $B=I_n - \frac{1}{n}J$. Then it's easy to see the conditions are satisfied and $X^{\top}(I_n - \frac{1}{n}J)X/(n-1)=s^2$ is independent of $(1/n,\ldots,1/n)^{\top} X=\bar{X}$.

Answer 3

There is a one-line proof, which, however, uses rather involved notions.

Namely, assuming $\sigma^2$ to be a known parameter, $\overline X$ is a boundedly complete sufficient statistic, while $\sum_{i=1}^n \frac{(X_i-\overline X)^2}{\sigma^2} \simeq \chi^2_{n-1}$ is an anciliary statistic. Therefore, by Basu's theorem, they are independent, and so are $\frac{\overline X - \mu}{\sigma}$ and $\sum_{i=1}^n \frac{(X_i-\overline X)^2}{\sigma^2}$.