6

I was having a discussion with a colleague today about correlation coefficients, and I was told that correlation coefficient between 2 random variables $X$ and $Y$ is proportional to the dot product of the two random variables.

I asked him what he means by this, and I was told that you can view random variables as vectors. I don't think I agree with that, but I don't have a sufficient background to really argue my point, but now I want to revisit this.

How can a random variable be viewed a vector? What is meant by dot product between 2 random variables -- is this actually formal terminology or something loosely used?

user5965026
  • 677

4 Answers4

6

The space $L^0(\Omega)$ of all random variables on a fixed sample space $\Omega$ is a vector space - the (outcome-wise) sum of two random variables is a random variable, and a scalar multiple of a random variable is again a random variable. So in that sense, random variables can be viewed as "vectors" because they are the elements of a vector space.

By "dot product" they likely mean the $L^2$ inner product, defined by $\langle X, Y \rangle = E[XY]$. This obeys the same basic algebraic properties as the ordinary Euclidean dot product: bilinear (with respect to the addition and scalar multiplication described above), symmetric, positive definite. Strictly speaking, this inner product doesn't necessarily live on $L^0(\Omega)$, but rather on the vector subspace $L^2(\Omega) \subset L^0(\Omega)$ consisting of random variables with finite second moment.

Nate Eldredge
  • 97,710
  • To clarify, would this mean if I took 50 samples of $X$ and 50 samples of $Y$ and arranged these into vectors each of length 50, then the expected value of the dot product $X^T Y$ would be $50 \sigma_{XY}$? Or $49\sigma_{XY}$? – Max Jun 26 '20 at 01:18
  • 1
    @Max: That is true, but it has nothing do with what I am saying here. The $L^2$ inner product is not in any way the same as the Euclidean dot product; it just has similar algebraic properties. – Nate Eldredge Jun 26 '20 at 01:22
  • 2
    E(XY) is indeed a dot product but it is equal to Cov(X,Y) + E(X)E(Y). For Cov(X,Y) to be a dot product you must restrict the space to random variables with zero mean and finite variance. And for Corr(X,Y) to be a dot product the variance must also be one. – phaedo Jun 26 '20 at 01:26
  • @phaedo Is it equal specifically to the dot product or to the inner product? The inner product is a generalization of the dot product, so if it's only defined to be equal to the inner product, it doesn't necessarily mean it's equal to the dot product right? – user5965026 Jun 26 '20 at 02:05
  • @user5965026 you can say inner product if you want, the only sensible way to define a dot or inner product between two RVs assimilated as vectors would be E(XY) – phaedo Jun 26 '20 at 02:25
5

For two joint discrete variables, the expectation of their product is a weighted dot product of their value vectors (all diagonal values are positive making the diagonal matrix positive definite):

$$ \mathbf{E}[XY] = \sum_{i=1}^n p_i x_i y_i = (x_1,...,x_n) \begin{pmatrix} p_1 & ... & 0\\ \vdots & \ddots & \vdots \\ 0 & ...& p_n \end{pmatrix} (y_1,...,y_n)^T$$

Here, $(X,Y)$ has $n$ possible realizations $(x_i, y_i)$ with probabilities $p_i$, $i=1,...,n$.

ir7
  • 6,249
  • 1
    This is quite intuitive for me. I wasn't thinking about the fundamental definition of discrete expectation because the case that I was discussing was such that the probabilities of each sample were equal (so basically instead of that probability matrix, we have a $\frac{1}{n}$ in front of the summation). But this makes it very clear where the dot products are coming from. – user5965026 Jun 26 '20 at 02:27
  • I’m glad it helps. – ir7 Jun 26 '20 at 02:32
2

A multivariate random variable can be considered as a random vector.

But correlation between two such random vectors (or more precisely, cross-correlation) would typically produce a matrix rather than a scalar value

My guess is that you may have been discussing two univariate random variables, say $X$ and $Y$, and calculating the sample correlation between them. If the sample size is $n$ then you could regard the two samples as random vectors $\mathbf{X}=(X_1,X_2,\ldots,X_n)$ and $\mathbf{Y}=(Y_1,Y_2,\ldots,Y_n)$. The sample correlation coefficient would then be $$\frac{\sum\limits_{i=1}^n (x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum\limits_{i=1}^n (x_i-\bar{x})^2 \sum\limits_{i=1}^n (y_i-\bar{y})^2}}$$ but you could calculate this using dot products and scalar arithmetic with the vector $\mathbf{1}_n$ of $n$ ones, with $$\frac{\mathbf X \cdot \mathbf Y - n(\mathbf X \cdot \mathbf 1_n)(\mathbf Y \cdot \mathbf 1_n) }{\sqrt{(\mathbf X \cdot \mathbf X - n(\mathbf X \cdot \mathbf 1_n)^2)(\mathbf Y \cdot \mathbf Y - n(\mathbf Y \cdot \mathbf 1_n)^2)}}$$

If you know that the expected values of $X$ and $Y$ are zero then you can use $$\frac{\sum\limits_{i=1}^n x_i y_i}{\sqrt{\sum\limits_{i=1}^n x_i^2 \sum\limits_{i=1}^n y_i^2}} \text{ or }\frac{\mathbf X \cdot \mathbf Y }{\sqrt{(\mathbf X \cdot \mathbf X)(\mathbf Y \cdot \mathbf Y )}}$$ and in this sense you might be stretching things and the correlation is proportional to the sample covariance $\mathbf X \cdot \mathbf Y$

Henry
  • 157,058
  • Right we were talking strictly about univariate RVs. The discussion was inspired by a derivation for least squares, where we observe that $X^Tr = 0$. $X$ is the regressor matrix and $r$ are the residuals. – user5965026 Jun 26 '20 at 02:11
  • This equality apparently suggests that the correlation between each regressor $X_i$ and the residual $r$ is zero, but I can't figure out why this is the case. The equality indicates that the dot product $\langle X_i, r \rangle $ between each regressor and $r$ is zero, but I'm not sure why this means their correlation is zero. I know that the mean of $r$ is zero, but unless the mean of $X_i$ is also zero, then I don't see why the correlation or covariance equates to zero as shown by your formula above. To get zero correlation we would need $\langle X_i - \mu_{X_i}, r \rangle = 0$ – user5965026 Jun 26 '20 at 02:11
  • Never mind, I figured it out now. – user5965026 Jun 26 '20 at 02:55
2

Suppose you have a collection of $n$ samples of dependent (in general) variables $X$ and $Y$: $(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)$

Then we can view this collection of $n$ samples as a pair of vectors in $\mathbb{R}^n$: $(x_1, x_2, \ldots, x_n)$ and $(y_1, y_2, \ldots, y_n)$.

Then what your colleague is saying is that we can view correlation between $X$ and $Y$ as a kind of normalized inner product between these two vectors.

Brian Tung
  • 34,160
  • This is exactly the situation we're talking about. I was originally looking at a derivation for least squares, where we see $X^Tr = 0$. $X$ is the regressor matrix and $r$ are the residuals. Apparently this suggests that each regressor $X_i$ and $r$ are uncorrelated, but I can't see why. Using @Henry formula for covariance above, this uncorrelation would be true if mean of the residuals and mean of the regressor are zero. The former is true for least squares, but the latter isn't necessarily true, so I don't quite see how $X^Tr = 0$ suggests that the regressors and residuals are uncorrelated. – user5965026 Jun 26 '20 at 02:16