8

I understand that a Markov chain involves a system which can be in one of a finite number of discrete states, with a probability of going from each state to another, and for emitting a signal.

Thus,an $N \times N$ transition matrix and an $N \times N$ emission matrix of real numbers adequately describe a Markov chain with $N$ states and $M$ emissions.

Is it possible to have a Markov chain with an infinite number of states? For example, if $N=2$ is a LED that can glow blue or red, $N=\infty$ would be a LED which can glow a color that is any mixture of blue or red.

Can't an infinitely-large matrix be represented by a function of two variables (the two indices)?

Rod Carvalho
  • 2,991
Superbest
  • 3,400

2 Answers2

7

Yes, Markov processes with infinitely many states are indeed considered. Random walks are a common example. The term "Markov chain" is often reserved for the case of a discrete state space. If the state space is finite, it's a "finite Markov chain". See e.g. http://www.statslab.cam.ac.uk/~james/Markov/

Robert Israel
  • 448,999
7

If the state space $S$ is countably infinite, finite sums are replaced by sums of convergent series and square matrices of finite size by arrays indexed by $S\times S$, but many formulas are unchanged, at least formally.

By contrast, here is a famous example of a Markov chain on an uncountable state space, namely, on $S=[0,1]$. Let $(U_n)_{n\geqslant1}$ and $(V_n)_{n\geqslant1}$ denote two independent i.i.d. sequences with $U_n$ uniform on $[0,1]$ and $V_n$ uniform on $\{0,1\}$, and let $X_0$ denote any random variable with values in $[0,1]$, independent of $(U_n)_{n\geqslant1}$ and $(V_n)_{n\geqslant1}$. One defines $(X_n)_{n\geqslant0}$ recursively, for every $n\geqslant0$, by $$X_{n+1}=U_nX_n+(1-U_n)V_n$$ Then $(X_n)_{n\geqslant0}$ is a Markov chain on $[0,1]$, whose stationary distribution can be computed explicitly.

In words, $V_n$ describes the decision to choose the interval $I_n=[0,X_n]$ if $V_n=0$ or $I_n=[X_n,1]$ if $V_n=1$, then $X_{n+1}$ is uniformly distributed in this (random) interval $I_n$.

Did
  • 279,727