5

Basic Question

Is there an intuitive explanation of standard deviation in terms of Euclidean distance in $n$ dimensional space?

Longer Version of Question

To begin a more detailed sketch of my question, for simplicity let's just focus on the simple case of a discrete random variable that is uniformly distributed. In this case, the variance is given by the following formula, which I've abducted straight from Wikipedia:

$$ \frac1{n}\sum_{i =1}^n (x_i - \mu)^2$$

where $\mu$ is the mean. The standard deviation is then the square root of this. Now, I can't help noticing that the square root of the sum returns the euclidean distance from the vector $X = (x_1, x_2, \dots, x_n)$ to the vector $\vec \mu = (\mu, \mu, \dots, \mu)$. That is, the standard deviation can be expressed as:

$$ \frac1{\sqrt{n}}|X - \vec \mu |$$

So I wonder, is there any significant conceptual relationship between this distance $|X - \vec \mu |$ and standard devation or is this just a coincidence?

Even More Details...

I have looked up many explanations of standard deviation and its cousin variance. Here are some that I've seen already, each sort of following from the previous one:

  • We square the values before summing to get rid of the sign, which is obviously not important. This explanation is often criticised by hardcore statisticians and I can sort of see why: it doesn't explain why squaring beats taking the absolute value.
  • We square the values so that we pay a greater price for greater deviations. This explains why squaring beats taking absolute values. But why not raise to the power of $4$, or $6$, or any other even power before summing? What is so special about $2$?
  • The thing that is so special about $2$ is that it's the second moment of intertia, whereas the mean is the first moment, so mechanically it makes sense. I don't follow this. My intuition is totally OK with the mean: the point where, if I put my finger, the weights on either side will balance. But the second moment is harder for me to imagine physically like this.

Note, this is a question about intuition. I "understand" the mathematical formula at a shallow level: what all its terms mean, how to calculate it given a dataset. But I am not comfortable with my grasp on why this formula is "the best" one to use in so many applications e.g. the least squares method to fit data. I'm particularly confused as to why squaring is chosen as opposed to raising to some other even power e.g. $9234324$.

And this is where my intuition steps in and tries to provide an explanation that goes right back to the fundamental theorem of Pythagoras: euclidean distance. Here is my thought process: "The number $2$ is special. It's the unique power that makes Euclidean distance work. So maybe it's also the unique number that makes variance work." But then why the multiplying factor of $\frac1{\sqrt{n}}$? Is it just simply a case of: swallow it up and accept the definition, or can this intuition be resolved somehow?

Colm Bhandal
  • 4,649
  • Related: https://math.stackexchange.com/questions/1860579/why-work-with-squares-of-error-in-regression-analysis, https://math.stackexchange.com/questions/1621759/neural-network-cost-function-why-squared-error, https://math.stackexchange.com/questions/1872055/is-minimizing-the-squared-errors-optimal – Colm Bhandal May 11 '20 at 08:59

1 Answers1

5

There is certainly a very clear "conceptual relationship" between the standard deviation and Euclidean distance: If we treat the whole available sample (the $x_i$'s) as a vector, then the Euclidean distance is a measure of how much this vector deviates from the vector containing the mean value, which is "the center" of the population.

But the standard deviation attempts to measure how much a single observation, not the whole sample, deviates "on average" from the mean value. Ah, then, why are we dividing by $\sqrt {n}$ and not by $n$?

Well, this becomes clear if we consider a vector $\mathbf x = (x,x,x,...x)$ Then, euclidean distance becomes

$$\sqrt {\sum_{i =1}^n (x_i - \mu)^2}=\sqrt n |x-μ|$$

So due to the square-root, Euclidean distance is not linearly additive, as we move from the one dimension, to $n$ dimensions : it does not increase by a factor of $n$, but only by a factor of $\sqrt {n}$. So to recover the "individual distance on average" we have to divide by $\sqrt n$.

Alecos Papadopoulos
  • 10,388
  • Great explanation. I do still worry a small bit about the "on average" bit. We're not exactly measuring "how much a single observation... deviates 'on average' from the mean value". Otherwise we'd be back at just taking the absolute value of the deviations, summing, and dividing by $n$ i.e. $\frac1{n}\sum_{i =1}^n |x_i - \mu|$. Still, this is a good intuitive explanation, thanks for sharing it. – Colm Bhandal Sep 04 '15 at 11:10
  • ...it seems to me like the standard deviation wants to measure how much the whole population vector deviates from its average, and then normalise this for the number of elements so that the special case of $(x, x, x, \dots, x)$ gives you back exactly the average deviation... – Colm Bhandal Sep 04 '15 at 11:18
  • @ColmBhadal We also do that, i.e. taking the average of the absolute deviation. It is called, ahem, "average absolute deviation". The use of second moments instead of the absolute deviations, has many arguments behind it, from mathematical convenience, to properties of statistical distributions, to the connection to Euclidean distance... your second comment has promise. – Alecos Papadopoulos Sep 04 '15 at 12:34
  • Yes I feel a bit more settled in my soul about the whole thing now, especially since I recently saw a result (not the proof though) of Gauss relating least squares to the normal curve. Making the second comment a bit more rigorous: We want to normalise the euclidean distance by multiplying it by a function of $n$ so that the result gives exactly the average absolute deviation for a "hyper-cubic" vector i.e. $(x, x, x, \dots, x)$. With this condition, by your above argument, the only solution is to multiply by $\frac1{\sqrt{n}}$. Yes I think I'm satisfied. For now. – Colm Bhandal Sep 04 '15 at 13:45