Confusion in stationary process and stationary distribution

Question

I have just started studying statistics, and have no background in this field(though I have a descent enough mathematical background).

I was studying about stationary distributions and stationary processes, and the book I am reading says that

A process is stationary if the joint distribution of random variables is same irrespective of time.

Following are my doubts regarding this:-

1. So does this mean that distribution(not joint) of all individual random variables(Xt) remains the same, independent of time?

2. I concluded from this that a stationary process is a process which has a stationary joint distribution of random variables, is this correct? If yes, then I guess it also implies stationary distribution of individual random variables(each Xt)?

3. Lastly, it says on wikipedia page of stationary process that "a stationary process is not the same thing as a "process with a stationary distribution"". This is the whole source of my confusion. A stationary process is something whose joint probability distribution doesn't change with time, so is there a difference between a stationary distribution and a distribution which doesn't change with time?(Is independent of time)

Though I have no background of statistics, I am really eager to learn, and would be really grateful if someone could please explain in simple terms,i.e. of a beginner. Also, it'd be great if you could suggest a book which will help in understanding these concepts(specifically of stochastic modelling) easily. Thanks a ton.

Answer 1

"Stationarity" is not a single concept. Viewing a stochastic process as a sequence of random variables (not necessarily a sequence in time, but along some index), then the two most commonly versions of the concept encountered are :

$1)$ The process is called strictly stationary if the joint distribution function of any specific number of elements of it is invariant to any shift in the structure of the indices. Namely say, $F(X_t, X_{t+2}) = F(X_{t+k}, X_{t+2+k})$, but not necessarily $F(X_t, X_{t+2}) = F(X_{t+k}, X_{t+2})$. Note that by construction a bivariate joint distribution is not identical to a trivariate joint distribution - so we can only compare bivariate with bivariate, trivariate with trivariate etc.

Strict stationarity does not require that the random variables involved have a density function (if they are continuous), or that if the density exists, it has finite moments. So strict stationarity does not imply...

$2)$ The process is called covariance-stationary (or "weakly" stationary, or "2nd-order" stationary) if the random variables involved all have
a) Constant and same unconditional mean (hence the 1st moment exists)

b) Constant and same unconditional variance (hence the 2nd moment exists, and the two produce the name "2nd-order" stationary)

c) The covariance between any two elements of the sequence may be non-zero, but it depends only on the "distance" (in terms of the index) between the two, and not on the value of the index itself (hence "covariance stationary").

Note that covariance-stationarity does not imply strict stationarity: nothing is said about the joint distribution, only about the first two moments. Higher moments may differ as the index evolves, and so the joint distribution won't be the same.

So a process may be strictly stationary but not covariance stationary (because the moments may not exist), or it may be covariance-stationary but not strictly stationary (because higher moments may differ).

There are more, like "rth-mean stationarity", etc, but as I said the above two are probably the most used.

Answer 2

There is a slight confusion with terminology here. People sometimes call a whole family of Markov processes $\{X_\mu(t)\}_{t\in I}$ a process itself ($I$ denotes some index set, for example, $\mathbb{R}$ or $\mathbb{N}$). That is, for every fixed $\mu$, $\{X_\mu(t)\}_{t\in I}$ is a Markov process and for all $\mu$s, $\{X_\mu(t)\}_{t\in I}$ has the same transition probabilities.

The "parameter" $\mu$ is used to describe the "starting" probability distribution of $\{X_\mu(t)\}_{t\in I}$. That is, suppose that $\{X_\mu(t)\}_{t\in I}$ takes values in a metric space $Y$, the process starts at some time $t_0\in I$, and let $\mathcal{B}(Y)$ denote the Borel sigma algebra of $Y$, then $\mu:\mathcal{B}(Y)\to[0,1]$ and

$$P(X_\mu(t_0)\in B)=\mu(B),\quad\quad\forall B\in\mathcal{B}(Y).$$

Then $\{X_\mu(t)\}_{t\in I}$ is said to have a stationary distribution $\pi$, if $\{X_\pi(t)\}_{t\in I}$ is a stationary process (stationary as in your definition). In short:

-1. Yes.

-2. Yes, if by "stationary joint distribution of random variables" you mean that the joint distribution does not depend on time, that is

$$P(X(t_1+h)\in B_1,\dots,X(t_n+h)\in B_n)=P(X(t_1)\in B_1,\dots,X(t_n)\in B_n),$$

for all Borel sets $B_1,\dots,B_n$, $t_1,\dots,t_n\in I$, $h$ such that $t_i+h\in I$ and $n\in\mathbb{N}$.

-3. See the above.