0

I am trying to show that $$\frac{1}{n} \sum_{i = 1}^n (y_i - \overline{y})(\hat{y}_i - \overline{\hat{y}}) \geq 0$$ where $y_i$s are the observations, $\hat{y}_i$s are corresponding LS fitting values, and $\overline{y}, \overline{\hat{y}}$ are the corresponding mean.

This is how I approached it: $$\begin{align*} \frac{1}{n} \sum_{i = 1}^n (y_i - \overline{y})(\hat{y}_i - \overline{\hat{y}}) &= \frac{1}{n}\textbf{Y}_c^T \hat{\textbf{Y}_c}\\ &= \frac{1}{n}(\textbf{CY})^T (\textbf{C}\hat{\textbf{Y}})\\ &= \frac{1}{n} \textbf{Y}^T\textbf{C}^T\textbf{C}\hat{\textbf{Y}}\\ &= \frac{1}{n} \textbf{Y}^T\textbf{C}\hat{\textbf{Y}}\\ &= \frac{1}{n} \textbf{Y}^T\textbf{C}\textbf{P}_x\textbf{Y}\\ &= \frac{1}{n} \textbf{Y}^T\textbf{C} \textbf{X}(\textbf{X}^T\textbf{X})^{-1}\textbf{X}^T\textbf{Y} \end{align*}$$ where $\textbf{C}$ is the centering matrix, $\textbf{P}_x$ is the projection matrix; if $\textbf{X}$ is full column rank, $\textbf{P}_x = \textbf{X}(\textbf{X}^T\textbf{X})^{-1}\textbf{X}^T$.

I tried to show the matrix $\textbf{C} \textbf{X}(\textbf{X}^T\textbf{X})^{-1}\textbf{X}^T$ is positive semidefinite but not exactly getting there. I feel like I've missed something obvious.

Any help is appreciated. Thanks all in advance.

Gavin
  • 553

1 Answers1

1

According to the wiki page you linked, $C$ is PSD, and $X(X^TX)^{-1}X^T$ is also PSD since for any $u \in \mathbb{R}^n$,

$$ u^T X(X^TX)^{-1}X^T u = y^T S^{-1} y = \|y\|^2_{S} \ge 0 $$

where $y := X^T u$ and $S = X^T X$, and the inequality follows since the preceding term is a (Mahalanobis) norm, and norms are always non-negative. The desired result follows from recalling that the matrix product of two PSD matrices is itself PSD, so

$$ C X(X^TX)^{-1}X^T \succeq 0 $$

Updating to include explanation of last statement: Note that I am assuming that matrices that are PSD are also symmetric - perhaps this is non-standard, so I am making it explicit. Both matrices being considered here are symmetric so it doesn't change the answer. For a proof of the fact that the product of two (symmetric) PSD matrices is PSD, see for example this post

WeakLearner
  • 5,982