Partial derivative of sigmoid function with sum (sigma) input

Question

Definition of logistic sigmoid function is:

$$σ(t)=\frac{1}{1 + e^{-t}}$$

Differentiating this is very simple, It just requires simplification with exponent rule and differentiation with quotient rule.

$$\frac{d}{dx}\frac{1}{1 + e^{-t}}=\frac{d}{dx}\frac{e^{t}}{1 + e^{t}}=\frac{e^{t}(1 + e^{t}) -e^{2t}}{(1 + e^{t})^{2}}=\frac{e^{t}}{(1 + e^{t})^{2}}$$

---

But what if instead of $t$, the input was weighted sum $\sum{w_ix_i}$ ?

The partial derivative with respect to $w_i$ somehow becomes $σ*1-σ*x_{i}$, this is how one person calculated it:

$$\frac{\partial}{w_{i}}\frac{1}{1 + e^{-\sum{w_ix_i}}}=\frac{-1}{(1+e^{-\sum{w_ix_i}})^{2}}*e^{-\sum{w_ix_i}}*(-x_i)$$ $$\frac{\partial}{w_{i}}\frac{1}{1 + e^{-\sum{w_ix_i}}}=\frac{1}{(1+e^{-\sum{w_ix_i}})}*\frac{e^{-\sum{w_ix_i}}}{{(1+e^{-\sum{w_ix_i}})}}*x_i$$

Now, considering that $1-σ$ is $\frac{e^{-\sum{w_ix_i}}}{{(1+e^{-\sum{w_ix_i}})}}$, we get:

$$\frac{\partial}{w_{i}}\frac{1}{1 + e^{-\sum{w_ix_i}}}=σ*1-σ*x_i$$

Is this derivative properly calculated? If so how exactly? $x_i$ is somehow outside of summation, is it due to it being treated as constant?

Answer 1

You have a composite function, $\sigma=1/(1-e^{-t})$ where $t=\sum w_i x_i$, so just use the chain rule: $$ \frac{\partial \sigma}{\partial w_i} = \frac{d \sigma}{d t} \cdot \frac{\partial t}{\partial w_i} = \sigma (1-\sigma) \cdot x_i . $$ (See here for a slick way of deriving $\frac{d \sigma}{d t}=\sigma (1-\sigma)$.)