Finding the gradient of the restricted function in terms of the gradient of the original function

Question

The following question showed up as part of a proof that I am doing for my research thesis.

If we have a differentiable function $f: \mathbb{R}^n \to \mathbb{R}$ and then set $n-d$ coordinates to zero we get a new differentiable function $g: \mathbb{R}^d \to \mathbb{R}$. Now, given the gradient $\nabla_x f(x)$, how one can get $\nabla_y g(y)$?

---

My try

Let $x \in \mathbb{R}^n$ and $S \subset \{1,\dots,n\}$ such that $|S|=d$ where $|\cdot|$ is the cardinality of the set. Let $U_S$ be a restricted identity matrix such that the $j$-th entry of the diagonal matrix is maintained if $j \in S$ otherwise it is set to zero. Also, let $I_S$ be the restriction of $U_S$ where we keep nonzero columns and remove zero columns. Hence,

$$ g(y)=f(U_Sx) $$ where $y=I_S^{\top}x$.

The above is the translation of what I stated in terms of functions $f$ and $g$.

From this point things are a little bit unclear. I think the answer should be $\nabla_y g(y)=I_S^{\top} \nabla_x f(x)$ but I do not know how to get it.

Also, I know using the chain rule $J_x f(U_S x)=J_{W} f(W)J_x W= J_{W} f(W)U_S$ where $J$ is the Jacobian and $W=U_S x$. In addition, $\nabla^{\top}_x f(U_Sx) = J_x f(U_S x)=J_{W} f(W)U_S$. I do not know how to put things together.

Answer 1

Since no one has posted an answer yet, and I get the same result as you suggest, I thought I'll post my solution for you to judge:

We have that $$U_S x = I_S y$$ so that, vieweing matrices as linear transformations $$g(y) = f(U_S x) = f(I_S y) = f\circ I_S (y)$$ And similar to what you write about $J_{U_S}(x)$ we have $J_{I_S}(y) = I_S$. Applying the chain rule: $J_{h_1 \circ h_2}(a) = J_{h_1}(h_2(a))J_{h_2}(a)$ then gives $$\begin{align} (\nabla_y g(y))^T = J_g(y) =\\ J_{f\circ I_S}(y) = \\ J_f(I_s y)J_{I_S}(y) = \\ J_f(U_s x)I_S = \\ (\nabla_x f(U_S x))^TI_S \implies \\ \nabla_y g(y) = [(\nabla_x f(U_S x))^TI_S]^T = I_S^T\nabla_x f(U_S x) \end{align} $$

Due to the definitions of $U_S$ and $I_S$, the zero columns in $I_S^T$ exactly matches the rows where $\nabla_x f(U_S x)$ and $\nabla_x f(x)$ might differ, so finally we obtain $$ \nabla_y g(y) = I_S^T\nabla_x f(U_S x) = I_S^T \nabla_x f(x) $$

Edit:
As pointed out in the comments, it would be more correct to write $$ \nabla_y g(y) = I_S^T\nabla_x f(I_S y) $$

Answer 2

Let fat matrix ${\bf S} \in \Bbb R^{d \times n}$ be

$${\bf S} := \begin{bmatrix} {\bf I}_d & {\bf O} \end{bmatrix} {\bf P}$$

where ${\bf P}$ is an $n \times n$ permutation matrix. Note that

$${\bf S} {\bf S}^\top = \begin{bmatrix} {\bf I}_d & {\bf O} \end{bmatrix} \underbrace{\,{\bf P} {\bf P}^\top}_{= {\bf I}_n} \begin{bmatrix} {\bf I}_d \\ {\bf O} \end{bmatrix} = {\bf I}_d$$

Let vector fields $\rho : \Bbb R^n \to \Bbb R^d$ and $\eta : \Bbb R^d \to \Bbb R^n$ be defined by

$$\rho := ({\bf x} \mapsto {\bf S} {\bf x}), \qquad \eta := ({\bf y} \mapsto {\bf S}^\top {\bf y})$$

and note that $\rho \circ \eta = \mbox{id}_{\Bbb R^d}$. Colloquially, if one "expands" and then "restricts", one ends up exactly where one started.

Given differentiable scalar field $f : \Bbb R^n \to \Bbb R$, let scalar field $g : \Bbb R^d \to \Bbb R$ be defined by

$$g := f \circ \eta$$

Hence,

$$\begin{aligned} g \left( {\bf y} + {\rm d} {\bf y} \right) = f \left( {\bf S}^\top {\bf y} + {\bf S}^\top {\rm d} {\bf y} \right) &= f \left( {\bf S}^\top {\bf y} \right) + \left\langle \nabla f \left( {\bf S}^\top {\bf y} \right) , {\bf S}^\top {\rm d} {\bf y} \right\rangle \\ &= f \left( {\bf S}^\top {\bf y} \right) + \left\langle {\bf S} \, \nabla f \left( {\bf S}^\top {\bf y} \right) , {\rm d} {\bf y} \right\rangle \end{aligned}$$

and, thus, the gradient of $g$ is

$$\nabla g \left( {\bf y} \right) = \color{blue}{{\bf S} \, \nabla f \left( {\bf S}^\top {\bf y} \right)}$$

or, more succinctly,

$$\boxed{ \qquad \\ \qquad \nabla g = \color{blue}{\rho \circ \nabla f \circ \eta \qquad \\ \qquad}}$$

Answer 3

$ \def\p{\partial} \def\L{\left}\def\R{\right} \def\LR#1{\L(#1\R)} \def\trace#1{\operatorname{Tr}\LR{#1}} \def\qiq{\quad\implies\quad} \def\grad#1#2{\frac{\p #1}{\p #2}} $For typing convenience, name the gradient $$\eqalign{ p = \grad{f}{x} \\ }$$ and rename the matrices $I_S\to S$ and $U_S\to U$.

Also note that $\,U=SS^T\,$ and that $$y=S^Tx \qiq dy=S^Tdx$$ Write the differential of the function and rearrange it to recover the desired gradient. $$\eqalign{ g(y) &= f(Ux) \\ dg &= df \\ &= p:d(Ux) \\ &= p:\LR{U\,dx} \\ &= p:\LR{SS^T\,dx} \\ &= \LR{S^Tp}:\LR{S^T\,dx} \\ &= \LR{S^Tp}:dy \\ \grad{g}{y} &= S^Tp \\ }$$

---

In the preceding, a colon is used to denote the Frobenius product, which is a concise notation for the trace $$\eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{A^TB} \\ A:A &= \big\|A\big\|^2_F \\ }$$ The properties of the underlying trace function allow the terms in a Frobenius product to be rearranged in many different but equivalent ways, e.g. $$\eqalign{ A:B &= B:A \\ A:B &= A^T:B^T \\ C:AB &= CB^T:A = A^TC:B \\ }$$