7

According to Wikipedia with a little rephrasing:

A state $i$ is transient if and only if $P(T_i < \infty) <1$, recurrent if and only if $P(T_i < \infty) =1$, where $T_i$ is the first hitting time to i, i.e. $T_i=\inf\{n \in \mathbb{N} \cup \{ \infty \}: X_n=i \mid X_0=i \}.$

If I understand correctly, this can be used as the definition of transient/recurrent state.

Usually $P(T_i < \infty)$ is written as a series $\sum_{n \in \mathbb{N}} P(T_i = n)$. But I would like to learn other ways to tell if a state is recurrent/transient, which might be easier in some cases.

  1. For example, can a transient/recurrent state be completely characterized in terms of closed subsets of states (defined similarly as an absorbing state), as follows (my own quote)?

    State $i$ is transient if and only if there exists a closed subset $S$ of states, s.t. $i \notin S$ and there exists $s \in S$ and $n \in \mathbb{N}$ and the $n$-step transition probability $p_{is}^{(n)} > 0$.

    Similarly, State $i$ is recurrent if and only if there does not exist such a closed subset of states as described above?

    Can we also characterize positive/null recurrence in terms of closed subsets of states?

  2. Off the top of your head, what are some other necessary and/or sufficient conditions for recurrent/transient and positive/null recurrent state?

Thanks and regards!

Tim
  • 47,382

2 Answers2

8

(0) The definition of $T_i$ on Wikipedia is awful on at least three counts. One should define $T_i=\inf A_i$ with $A_i=\{n\ge1:X_n=i\}\cup\{+\infty\}$ and one should say that $i$ is transient if and only if $P(T_i<+\infty|X_0=i)<1$ and recurrent otherwise.

(1) In some cases there exists no closed subset at all and the existence of closed sets is not related to recurrence or transience.

First example: consider a homogenous random walk on $\mathbb{Z}$. Thus $p_{n,n+1}=a$ and $p_{n,n-1}=1-a$ for every $n$, for a given $a$ in $(0,1)$. Then there exists no closed set except $\mathbb{Z}$ and the chain is recurrent if $a=\frac12$ and transient otherwise.

Second example: consider a homogenous birth-and-death chain such that $0$ is absorbing. Thus $p_{0,0}=1$ and $p_{n,n+1}=a$ and $p_{n,n-1}=1-a$ for every $n\ge1$, for a given $a$ in $(0,1)$. Then the set $S=\{0\}$ is closed and the chain is recurrent if $a\le\frac12$ and transient otherwise.

(2) You could (should?) read the beautiful small book Random Walks and Electric Networks by Peter G. Doyle and J. Laurie Snell, which explains this and a lot of related stuff in a very accessible way.

Did
  • 279,727
  • @Didier: Thanks! (1) Does $\inf{n\in\mathbb{N}^:X_n=i}\cup{+\infty}$ mean to take infimum of ${n\in\mathbb{N}^:X_n=i} \cup { +\infty }$ ? If yes, is it same as $\inf{n\in\mathbb{N}^* \cup { \infty }:X_n=i}$? What is the difference between $\mathbb{N}^*$ and $\mathbb{N}$? (2) if there exists some closed proper subset, are my original statements true? If not, how do they become false? (3)if there does not exist any closed proper subset, how do my original statements become false? Thanks for elaboration! – Tim Mar 18 '11 at 13:56
  • @Tim See edited answer. – Did Mar 18 '11 at 14:20
  • @Didier: Thanks! (1) You wrote ${n\geq 1:X_n=i}\cup {+\infty}$ instead of ${n \in \mathbb{N} \cup {+\infty}:X_n=i}$. Is it because the index/parameter set of the stochastic process ${ X_n}$ is $\mathbb{N}$, not $\mathbb{N} \cup {+\infty}$? (2) Does the first hitting time of state $i$ assume starting at $i$? This seems to be no in your definition. If starting at $i$ is needed for the definition of the first hitting time of $i$, how to write its definition? Is $T_i=\inf A_i$ with $A_i={n\ge1:X_n=i \mid X_0=i}\cup{+\infty}$ correct? If not how is it wrong? Thanks! – Tim Mar 18 '11 at 17:42
  • 1
    @Didier, (+1), especially for the Doyle and Snell remark. I didn't know that this little gem of a book was that well known. That definitely ought to be "should" in your answer! – cardinal Mar 18 '11 at 17:46
  • @Tim (1) Yes. Note that $X_n$ is not even defined for $n=+\infty$. (2) The first hitting time of $i$ is defined for every initial distribution of the process but recurrence and transience involve the properties of $T_i$ for the process starting at $i$. The definition of $T_i$ should be written as I wrote it... The definition of $A_i$ you suggest does not make sense since $A_i$ is a (random) subset of $\mathbb{N}^\cup{+\infty}$ and the sign $|$ means conditionally on* as in $P(B|A)$ or in $E(\xi|A)$ for an event $B$ or an integrable random variable $\xi$. .../... – Did Mar 18 '11 at 18:03
  • .../... So what would be ${n\ge1:X_n=i|X_0=i}$? (The author of this wikipedia page should be hanged...) To indicate a conditioning by the event $[X_0=i]$, one can write $P(B|X_0=i)$ or $P_i(B)$ or $E(\xi|X_0=i)$ or $E_i(\xi)$. – Did Mar 18 '11 at 18:03
  • @cardinal Glad to see we meet on this as well. Dunno if the book is so well known, generally speaking. – Did Mar 18 '11 at 18:05
  • @Didier: Thanks! I know you hate the way he wrote it, but how about $A_i={n\ge1:X_n=i, X_0=i}\cup{+\infty}$ if someone insists on requiring starting at state $i$? – Tim Mar 18 '11 at 18:25
  • @Tim: Sorry but the answer is no. All I can suggest is that you read slowly my comment and try to digest it, since I cannot do better than what I wrote. (Or find yourself a good introductory probability book, for example here http://www.math.duke.edu/~rtd/.) – Did Mar 18 '11 at 18:41
6

  1. Tim's characterization of states in terms of closed sets is correct for finite state space Markov chains. Partition the state space into communicating classes. Every recurrent class is closed, but no transient class is closed (because the chain must eventually get "stuck" in some recurrent class). The part in parentheses is false for infinite state space chains, as Didier's answer shows.

  2. Another well-known characterization is that a state $i$ is transient if and only if $$\sum_{n=1}^\infty P(X_n=i | X_0=i)<\infty.$$ This criterion is used, for example, to prove Polya's result that the symmetric random walk on $\mathbb{Z}^d$ is recurrent if $d=1,2$, but transient when $d\geq 3$.

  3. Similarly, the probability
    $$P(X_n=i\mbox{ for infinitely many } n | X_0=i)$$ is equal to zero or one, depending on whether the state $i$ is transient or recurrent.

  • 1
    Thanks! (1) is the condition in your third part a necessary and sufficient condition for transient or recurrent? (2) what are some other ways to charaterize null/positive recurrence, besides the definition of whether $E(T_i)$ is infinite? For example, in terms of transition probabilities, and/or in terms of closed subset of states? Thanks! – Tim Mar 18 '11 at 17:33
  • @Tim Yes, both parts 2 and 3 give necessary and sufficient conditions valid for both finite and infinite state space chains. As for your other question, an irreducible chain is positive recurrent iff there is an probability vector $\pi$ that is invariant, i.e., $\pi=\pi P$. –  Mar 18 '11 at 17:40
  • Thanks! In the first part of your reply, for infinite state space, do you mean my charaterization is not necessary for transient/recurrent state? But is it sufficient? Thanks! – Tim Mar 18 '11 at 20:15
  • My bad. Didier's second example is a counterexample to the sufficient statement. I went out for a while before coming back, and I thought I had read his edit but I actually missed that. – Tim Mar 18 '11 at 23:09