1

Reading the justification for Least Squares Minimization (in Matrix form), I come up with the expression:

$$0=\frac{dS}{d\beta}\left( \hat{\beta}\right)= \frac{d}{d\beta}\left.\left( y^Ty-\beta^T X^Ty-y^TX\beta+\beta^TX^TX\beta \right)\right\rvert_{\beta=\hat{\beta}} = -2X^Ty+ 2X^TX\hat{\beta}$$

I am not sure I follow the rules they apply here, what are the main matrix derivative rules applied in this expression ?

Wikipedia Source

Tim
  • 767
Mah Neh
  • 129
  • This might help https://math.stackexchange.com/a/189436/27978 – copper.hat Jul 23 '23 at 04:11
  • Let $f(\beta) = |X\beta-y|^2$ and expand using the inner product. Then compute $f(\beta+h)-f(\beta) = (2 \beta^TX^T X - 2 y^TX )h+h^T X^T X h$ and note that the last term is of order $(|h|^2)$. – copper.hat Jul 23 '23 at 04:14
  • @copper.hat no idea what you mean – Mah Neh Jul 23 '23 at 04:22
  • You might want to review what a matrix derivative means. – copper.hat Jul 23 '23 at 04:23
  • I get that you're adding a tiny matrix h to b, and that would be equivalent to a differential when h goes to 0. But the distributive of the dot product is confusing, you reduced it to a single step @copper.hat – Mah Neh Jul 23 '23 at 04:24
  • With the linear operator $L(h) = (2 \beta^TX^T X - 2 y^TX )h$ we see that $f(\beta+h)-f(\beta)-L(h) $ is $o(|h|)$, hence $L$ is the derivative. – copper.hat Jul 23 '23 at 04:27
  • so in matrices the derivative $ lim_{h\rightarrow 0} \Delta f / \Delta h$ is rather $lim_{h\rightarrow 0} \Delta f = f_2*\Delta h$ in very broad terms ? @copper.hat – Mah Neh Jul 23 '23 at 05:37

2 Answers2

1

https://en.wikipedia.org/wiki/Matrix_calculus

Check out the section "Scalar-by-vector identities"

Alan Chung
  • 1,182
0

They are of the form $α = x^T A x$, $α = y^T A x$ and what you need is to know how to compute $\frac{\partial α}{\partial x}$ and $\frac{\partial α}{\partial y}$.

The following referenece has exactly the form of what you want to take the derivative. You can also find how the derivative is proved.

References: MatrixCalculus

Proposition 7,8

Tim
  • 767