0

Suppose that $A$ is an $n\times k$ matrix with $n>k$. What vector $x\in \mathbb{R}^k$ minimizes the following quadratic form: $$ \begin{bmatrix} 1&x^{\top} \end{bmatrix}\begin{bmatrix} n & \boldsymbol{1}^{\top}A \\ A^{\top}\boldsymbol{1} & A^{\top}A \end{bmatrix}^{-1}\begin{bmatrix} 1 \\ x \end{bmatrix}? $$ Here $\boldsymbol{1}$ is a vector of ones. When $k=1$, it is easy to show that $x^*=A^{\top}1/n$ by explicitly inverting the matrix in the middle.

Robert W.
  • 722
  • What is the problem with your method when k does not equal 1? I worked out a solution and it is the same as what you have for any k. – dgadjov Oct 01 '21 at 20:26
  • @AsAnExerciseProve Thanks. I'll try to make those computations. – Robert W. Oct 01 '21 at 22:33
  • This is exactly what I did and there is a nice cancellation that gives you the answer Robert posted. – dgadjov Oct 02 '21 at 00:29
  • 1
    It might be helpful to note that $$ \pmatrix{n & \boldsymbol{1}^{\top}A \ A^{\top}\boldsymbol{1} & A^{\top}A} = \pmatrix{1 & 0\0 & A}^\top \pmatrix{n & \mathbf 1^\top\ \mathbf 1 & I_n} \pmatrix{1 & 0\0 & A} $$ – Ben Grossmann Oct 02 '21 at 00:30
  • @AsAnExerciseProve What formula did you use for the explicit inverse of the matrix in the middle? – Ben Grossmann Oct 02 '21 at 00:31
  • @BenGrossmann https://math.stackexchange.com/questions/182309/block-inverse-of-symmetric-matrices. It actually gives the result. – Robert W. Oct 02 '21 at 00:40

2 Answers2

1

The inverse of the mid matrix is given by $$ \begin{bmatrix} n & \boldsymbol{1}^{\top}A \\ A^{\top}\boldsymbol{1} & A^{\top}A \end{bmatrix}^{-1}=\begin{bmatrix} (\boldsymbol{1}^{\top}M_A\boldsymbol{1})^{-1} & -\frac{\boldsymbol{1}^{\top}A}{n}(A^{\top}M_{\boldsymbol{1}}A)^{-1} \\ -(A^{\top}M_{\boldsymbol{1}}A)^{-1}\frac{A^{\top}\boldsymbol{1}}n &(A^{\top}M_{\boldsymbol{1}}A)^{-1} \end{bmatrix}, $$ where $M_B=I-B(B^{\top}B)^{-1}B$. Differentiating the quadratic form w.r.t. $x$, one gets the first order condition: $$ -2(A^{\top}M_{\boldsymbol{1}}A)^{-1}\frac{A^{\top}\boldsymbol{1}}{n} +2(A^{\top}M_{\boldsymbol{1}}A)^{-1} x^*=0. $$ Thus, $$ x^*=\frac{A^{\top}\boldsymbol{1}}{n}. $$

Robert W.
  • 722
1

Let $a\in\mathbb{R},$ $b\in\mathbb{R}^k$ and $C\in\mathbb{R}^{k\times k}$ such that $$ \begin{bmatrix} n & \boldsymbol{1}^{\top}A \\ A^{\top} \boldsymbol{1} & A^{\top}A \end{bmatrix}^{-1} = \begin{bmatrix} a & b^{\top} \\ b & C \end{bmatrix} $$ Note that $C$ is symmetric and positive definite, because it is a principle submatrix of an invertible matrix that in turn is the product of a matrix and its transpose: $$ \begin{bmatrix} n & \boldsymbol{1}^{\top}A \\ A^{\top} \boldsymbol{1} & A^{\top}A \end{bmatrix} = \begin{bmatrix} \boldsymbol{1} & A \end{bmatrix}^{\top} \begin{bmatrix} \boldsymbol{1} & A \end{bmatrix} $$ Then $$ \begin{bmatrix} 1 \\ x\end{bmatrix}^{\top} \begin{bmatrix} n & \boldsymbol{1}^{\top}A \\ A^{\top} \boldsymbol{1} & A^{\top}A \end{bmatrix}^{-1} \begin{bmatrix} 1 \\ x\end{bmatrix} = x^{\top}C x + x^{\top}b + b^{\top}x + a \\ \phantom{x} \\ = (Cx+b)^{\top}C^{-1}(Cx+b)-b^{\top}C^{-1}b+a $$ Therefore, without knowing $a$, $b$ and $C$, we know that we get the minimum if $Cx+b=0,$ this means, when $x=-C^{-1}b.$

We do not need the formula for the inverse of a block matrix to proceed. From the definition of $a,$ $b$ and $C$, we get $$ \begin{bmatrix} n & \boldsymbol{1}^{\top}A \\ A^{\top} \boldsymbol{1} & A^{\top}A \end{bmatrix} \begin{bmatrix} a & b^{\top} \\ b & C \end{bmatrix} = \begin{bmatrix} * & n b^{\top}+ \boldsymbol{1}^{\top}A C\\ * & * \end{bmatrix} = I $$ which means $\;n b^{\top}+ \boldsymbol{1}^{\top}A C = 0\;$ or $\;bn + CA^{\top}\boldsymbol{1} = 0\;$ or $\;\frac{1}{n}A^{\top}\boldsymbol{1} = -C^{-1}b = x.$

Reinhard Meier
  • 7,331
  • 10
  • 18