I'm working through a problem regarding expected values in Markov chains, and at some point it says:
Recall from probability that if $X$ is a positive integer valued random variable, then $\mathbb{E}[X] = 1 + \sum^\infty_{k=1}\mathbb{P}(X > k)$.
I know that by definition $\mathbb{E}[X] = \sum_a a\mathbb{P}(X = a)$, but I can't see how the above equality follows from this, nor am I sure if this is how to approach the problem.
$aP(X = a) = P(X=a) + \dots + P(X = a)$ ($a$ times)
Now assuming $X$ takes only integer values $\ge 1$, you have
$$1 = P(X > 0) = P(X = 1) + \color{red}{ P(X = 2)} + \color{green}{P(X=3) }\dots$$ $$P(X > 1) = \color{red}{P(X = 2)} + \color{green}{P(X = 3) }+ \dots $$ $$P( X > 2) = \color{green}{P(X = 3)} + P(X = 4) + \dots $$
and so on.
Summing everything on the LHS you get $ 1 + \sum_{k=1}^\infty P(X > k)$, on the RHS you find $\sum_{a = 1}^\infty aP(X = a)$
Hence $$E[X] = \sum_{a = 1}^\infty aP(X = a) = 1 + \sum_{k=1}^\infty P(X > k)$$
Let $X$ be a nonnegative random variable with distribution function $F_X$. Then, by Fubini's Theorem, we have $$E[X]=\int_{[0,\infty)}\,x\,\text{d}F_X(x)=\int_{[0,\infty)}\,\int_{[0,\infty)}\,\chi_{[0,x)}(t)\,\text{d}t\,\text{d}F_X(x)=\int_{[0,\infty)}\,\int_{[0,\infty)}\,\chi_{[0,x)}(t)\,\text{d}F_X(x)\,\text{d}t\,,$$ where $\chi_E$ is the characteristic function on a set $E$. That is, $$E[X]=\int_{[0,\infty)}\,\int_{[0,\infty)}\,\chi_{(t,\infty)}(x)\,\text{d}F_X(x)\,\text{d}t=\int_{[0,\infty)}\,\text{Prob}(X> t)\,\text{d}t\,.$$ In particular, if $X$ is discrete, say, $X(\omega)\in\left\{x_1,x_2,\ldots\right\}$ for all $\omega$ with $0\leq x_1<x_2<\ldots$ and $\lim\limits_{n\to\infty}\,x_n=\infty$ (this ordering doesn't always exist, for example, when the possible values of $X$ are in $\mathbb{Q}_{>0}$), then $$\begin{align} E[X]&=\int_{[0,x_1)}\,\text{Prob}(X>t)\,\text{d}t+\sum_{i=1}^\infty\,\int_{[x_{i},x_{i+1})}\,\text{Prob}(X>t)\,\text{d}t \\&=\int_{[0,x_1)}\,1\,\text{d}t+\sum_{i=1}^\infty\,\int_{[x_{i},x_{i+1})}\,\text{Prob}(X>x_{i})\,\text{d}t \\ &=x_1+\sum_{i=1}^\infty\,\left(x_{i+1}-x_i\right)\,\text{Prob}\left(X>x_{i}\right)\,. \end{align}$$ If $\left\{x_1,x_2,\ldots\right\}=\{1,2,\ldots\}$, then we get $$E[X]=1+\sum_{i=1}^\infty\,\text{Prob}(X>i)\,,$$ as required. As Zachary Selk mentioned, if $\left\{x_1,x_2,\ldots\right\}=\{0,1,2,\ldots\}$, then $$E[X]=\sum_{i=0}^\infty\,\text{Prob}(X>i)\,.$$
This article in Wikipedia gives an answer to your question.
A slightly more general statement says the following:
When a random variable $X$ takes only values in nonnegative integers, we can use the following formula for computing its expectation (even when the expectation is infinite): $$ \operatorname{E}[X]=\sum\limits_{i=1}^\infty P(X\geq i)=P(X\geq 1)+\sum\limits_{i=1}^\infty P(X> i)\tag{*} $$ In the case when $X$ takes values in positive integers, one has $P(X\geq 1)=1$ in $(*)$.
There is an interesting geometric proof for this proposition given in Section $5$ of Probability and Measure by Patrick Billingsley. I will copy the illuminating figure from this textbook:
Note that the equation to be proved can also be written as $$E[X] = \int_0^\infty P[X \geq x] dx. \tag{1} $$
For your special case, $x_i$s in the figure are integers $i$, but they can be any finitely many real numbers. To get a proof from the above picture, the author explains:
There is for $(1)$ a simple geometric argument involving the "area over the curve." If $p_i =P[X = x_i]$, the area of the shaded region in the figure is the sum $$p_1x_1 + \cdots +p_k x_k = E[X]$$ of the areas of the horizontal strips; it is also the integral of the height $P[X \geq x]$ of the region.
