Probability limit of a non-aperiodic Markov chain

Question

We know that, for an irreducible and aperiodic discrete time Markov chain with stationary distribution $\pi$, we have $$\lim_{n \rightarrow \infty}p^n(x,y) = \pi(y)$$ However - what if the Markov chain is not aperiodic? Is there an easy way to calculate this probability limit, or does it not exist?

Answer 1

Let $p_{xy}^{(n)}$ and $f_{xy}^{(n)}$ be the n step transition probability from x to y and the first entry probability in n steps from $x$ to $y$. Let $\mu_y$ be the mean return time to $y$. Here is a general result

Theorem : Let $(X_n)_{n=0}^\infty$ be a homogeneous markov chain with discrete state space $E$.

(i) Suppose $y\in E$ is transient. Then for any $x\in E$, $\sum_{n=1}^\infty p_{xy}^{(n)}<+\infty$ and $\underset{n\to\infty}{\lim}p_{xy}^{(n)}=0$.

(ii) Suppose $y\in E$ is recurrent and aperiodic. Then for any $x\in E$, $\underset{n\to\infty}{\lim}p_{xy}^{(n)}=\begin{cases}0 & \mbox{if }\mu_y=+\infty \\ \frac{f_{xy}}{\mu_y} & \mbox{if } \mu_y<+\infty\end{cases}$.

(ii') Suppose $y\in E$ is recurrent and aperiodic and $x\in E$ such that $x\leftrightarrow y$. Then $\underset{n\to\infty}{\lim}p_{xy}^{(n)}=\begin{cases}0&\mbox{if }\mu_y=+\infty \\ \frac{1}{\mu_y}&\mbox{if }\mu_y<+\infty\end{cases}$

(iii) Suppose $y\in E$ is recurrent and periodic with period $d>1$. Then for any $x\in E$ and each $r=0,1,\cdots,d-1$, $$\underset{n\to\infty}{\lim}p_{xy}^{(nd+r)}=\begin{cases}0 & \mbox{if }\mu_y=+\infty \\ \frac{d}{\mu_y}\sum_{k=0}^\infty f_{xy}^{(kd+r)}&\mbox{if } \mu_y<+\infty\end{cases}$$

(iii') Suppose $y$ is recurrent and periodic with period $d>1$ and $x\in E$ such that $x\leftrightarrow y$ so $x,y$ belong to the same equivalence class $S$. Also suppose $x\in C_a$ and $y\in C_b$ where $a,b\in\{0,1,\cdots,d-1\}$ and $C_0,C_1,\cdots,C_{d-1}$ are cyclic subclasses of $S$ with respect to a fixed state in $S$. Then $$\underset{n\to\infty}{\lim}p_{xy}^{(nd+b-a)}=\begin{cases}0 & \mbox{if }\mu_y=+\infty \\ \frac{d}{\mu_y}&\mbox{if } \mu_y<+\infty\end{cases}$$

Corollary

(i) Let $y\in E$ be recurrent. Then $y$ is null recurrent if and only if $\underset{n\to\infty}{\lim}p_{yy}^{(n)}=0$.

(ii) Let $y\in E$ be recurrent. Then $y$ is positive recurrent if and only if $\underset{n\to\infty}{\varlimsup}p_{yy}^{(n)}>0$.

(i') If $y\in E$ is transient or null recurrent, then for any $x\in E$, $\underset{n\to\infty}{\lim}p_{xy}^{(n)}=0$.

(ii') If $y\in E$ is positive recurrent and aperiodic, then for any $x\in E$ such that $x\leftrightarrow y$, $\underset{n\to\infty}{\lim}p_{xy}^{(n)}=\frac{1}{\mu_y}>0$.

Note: In (i'), we don't have any requirement on the period of $y$ or whether $x$ communicates with $y$.

Answer 2

The chain does not converge to equilbrium, where equilibrium means that for all $x,y \in S$, $\lim \limits_{n \to \infty} p^n(x,y) $ exists and is independent of $x$.

Proof: Your chain is irreducible with period $3$, and therefore $\lim \limits_{n \to \infty} p^{3n+1}(x,x) = \lim \limits_{n \to \infty} p^{3n+2}(x,x) = 0$ for all $x \in S$. But $p^{3n}(1,1)=1$ for all $n$, so the limit $\lim \limits_{n \to \infty} p^n(1,1)$ does not exist.