3

My textbook, Introduction to Probability by Blitzstein and Hwang, says the following in a section on covariance and correlation:

Definition 7.3.1 (Covariance). The covariance between r.v.s $X$ and $Y$ is

$$\text{Cov}(X, Y) = E((X - EX)(Y - EY).$$

Multiplying this out and using linearity, we have an equivalent expression:

$$\text{Cov}(X, Y) = E(XY) - E(X)E(Y).$$

Let's think about the definition intuitively. If $X$ and $Y$ tend to move in the same direction, then $X - EX$ and $Y - EY$ will tend to be either both positive or both negative, so $(X - EX)(Y - EY)$ will be positive on average, giving a positive covariance. If $X$ and $Y$ tend to move in opposite directions, then $X - EX$ and $Y - EY$ will tend to have opposite signs, giving a negative covariance.

If $X$ and $Y$ are independent, then their covariance is zero. We say that r.v.s with zero covariance are uncorrelated.

  1. The intuition with regards to zero covariance is clear to me, since I assume that zero covariance implies that the random variables $X$ and $Y$ are independent, and so $E(XY) = E(X)E(Y)$, right?

  2. Why will $X - EX$ and $Y - EY$ tend to be either both positive or both negative if $X$ and $Y$ tend to move in the same direction?

  3. Why will $X - EX$ and $Y - EY$ tend to opposite signs if $X$ and $Y$ tend to move in opposite directions?

I would greatly appreciate it if people could please take the time to clarify this points.

Community
  • 1
The Pointer
  • 4,182
  • 2
    Your assumption, that uncorrelated implies independent, is wrong. – kimchi lover Oct 14 '19 at 14:00
  • 1
    @kimchilover Actually, I think the converse is true, right? That is, if $X$ and $Y$ are independent, then they're uncorrelated? – The Pointer Oct 14 '19 at 14:05
  • 1
    Bingo! One useful example is if $(X,Y)$ is uniformly distributed on the unit circle in the plane. They are definitely not independent, but they are uncorrelated. – kimchi lover Oct 14 '19 at 14:07
  • For your second and third questions, it may be helpful to consider $X$ and $Y$ with mean $0$. – Theoretical Economist Oct 14 '19 at 14:36
  • @ThePointer The correlation of independent variables is zero or undefined. – J.G. Oct 14 '19 at 14:49
  • I have always found this wikipedia article on correlation to be excellent in giving insights about positive vs zero vs negative correlation, esp. some of the pictures of examples. (BTW correlation is related to covariance via rescaling by standard deviations, and in any case, correlation and covariance have the same $\pm$ sign, which seems to be your main question.) – antkam Oct 14 '19 at 18:33

2 Answers2

1

This example may help you interpret what "move in the same direction" or "move in the opposite direction" might mean.

Let $(X,Y) = (1,1)$ with probability $1/2$, and $(X,Y) = (-1,-1)$ otherwise. Then, $\text{Cov(X,Y)} = E[XY] = 1$. In this example, $X$ and $Y$ move in the same direction together. (Indeed, we have that $X=Y$.)

Suppose instead that $(X,Y) = (1,-1)$ with probability $1/2$, and $(X,Y) = (-1,1)$ otherwise. Then, $\text{Cov}(X,Y) = E[XY] = -1$, and $X$ and $Y$ tend to move in opposite directions.

More generally (still restricting ourselves to mean zero random variables), when $X$ and $Y$ are positively correlated, so that they tend to move together in the same direction, then on average, whenever $X$ is positive then so is $Y$, and whenever $X$ is negative and so is $Y$ (and vice-versa), so that we have that $XY > 0$ on average. You can interpret negative correlation similarly.

If we want to extend this interpretation to random variables with non-zero mean, then consider the above intuition for the random variables $X-E[X]$ and $Y-E[Y]$.

Theoretical Economist
  • 6,472
1

I discussed in this answer to a question about standard deviations that non-constant finite-variance zero-mean variables (such as your $X-EX,\,Y-EY$) can be thought of as vectors, with the standard deviation being the length of that vector. The covariance is then the dot product of two vectors, and the correlation is the cosine of the angle between them. Imagine we gradually reduce the correlation from $1$ to $-1$ by "rotating" one variable (this can actually be made rigorous). When it's $1$, $0$ or $-1$, the vectors are parallel, orthogonal or antiparallel, respectively. A value close to $1$ means the vectors have similar directions, a value close to $-1$ means they're almost pointing in opposite directions, and so on. Hopefully, that addresses your second and third questions (comments have already addressed the first).

J.G.
  • 115,835