1

I am receiving a stream of numbers. I need to calculate the standard deviation of the last 100 numbers at each step. I don't have enough time for calculating it from scratch.

Is there a cheaper way of this calculation?

mustafa
  • 113
  • 5
  • 2
    Not sure what you are hoping for here. Even computing the running average requires you to retain the list: $\text {mean}(x_2,\cdots, x_{101})=\text {mean}(x_1,\cdots, x_{100})-\frac 1{100}x_1+\frac 1{100}x_{101}$. – lulu Jun 11 '18 at 13:52
  • 2
    This older Questioni deals with the variance of a stream of sample values. While the basic idea of keeping running sums of $x_i$ as well as $x_i^2$ is pretty obvious, there are some subtle issues in how to maintain numerical stability. Similar material can be found at the Data Signal Processing and CrossValidated SE sites. – hardmath Jun 11 '18 at 23:13
  • Please visit following: https://math.stackexchange.com/questions/2815732/calculating-standard-deviation-of-a-moving-window Very nice explanation. – Bhaumik Darji Apr 16 '20 at 12:42

1 Answers1

2

Assuming you are using SD with Bessel's correction, call $\mu_n$ and $SD_n$ the mean and standard deviation from $n$ to $n+99$. Then, calculate $\mu_1$ and $SD_{1}$ afterwards, you can use the recursive relation $\mu_{n+1}=\mu_n-\frac 1{99}x_n+ \frac 1{99}x_{n+100}$ and $SD_{n+1}=\sqrt{SD_n^2-\frac1{99}(x_n-\mu_n)^2+\frac1{99}(x_{n+100}-\mu_{n+1})^2}$.

  • thanks, this is a good enough approximation for a list of 100 elements from standard normal distribution – mustafa Jun 12 '18 at 10:25