Updating the variance of a sliding window without using stored data

Question

There is a very nice way to compute the variance of a moving window as detailed by Knuth and Cook and answered locally here, also on a blog here. The method requires you to make use of the data in the window in the form of $(x_N - x_0)$. I can't store the windowed data, but would still like to simulate the behaviour of having a windowed running update. Is there a way to maintain an estimate of the evolving variance using only the previous step parameters $\mu, \sigma^2, d^2$?

Essentially, the parameters give me an estimate for the distribution for the previous time step, then I see a data point, and if it is an "unexpected" point I want the variance to inflate slightly, and if it is "expected" I want the variance to shrink a bit. The mean and variance of my data stream moves around, but eventually converges, and I'd like the running estimate to capture this behaviour.

Here's a contrived example. $\mu$ starts at 0, linearly increases and is then flat at 5 again. $\sigma$ starts at 1, linearly increases and decreases back to 1. The blue line is computed using a window of 30 points. Green line is using the normal streaming equations, but as expected it is computing the params for the whole data sample, not just the window. The orange line is just capping $N = min(N, 30)$ in the normal streaming setting, but it does not adapt quickly enough, or forget the old samples. Any help would be much appreciated!

Answer 1

I don't think this is possible to do exact. You will need information about the point leaving the window: Consider the two instantiations of moving windows $a = [1, 3, 1, 3, 1, 3]$ and $b = [3, 1, 3, 1, 3, 1]$. Both have a mean of 2, and a variance of 1, but adding a new number (e.g. 2) and removing the oldest number will result in different means over the new windows. However, if you'd solely base your answer on the current mean and standard deviation, they will not differ.

A method you could try to approximate the mean and standard deviation is to do some kind of a running average. Here you would update your mean and standard deviation based on a new number $x$ via: $$\mu := (1-\rho) \mu + \rho x$$

$$\sigma^2 := (1-\rho) \sigma^2 + \rho (x -\mu)^2$$

For a (small) $\rho \in (0,1)$ that can be tuned. This should induce some latency, but when the distribution stays constant for a while, they should eventually converge to the correct numbers.

Answer 2

@student13's answer was just what I was looking for. Here are the updated graphs if anyone is interested, I used a $\rho = \frac{1}{15}$: