4

There are several variations of this Drunk man (or a monkey) at the cliff problem.

I wonder what will be the expected sum of rewards if the drunk man gets paid while he stays alive.

The setup is like this.

At period zero, a man is standing at $n$ step from the edge of a cliff. In each period from period $1$, he takes independent steps; $k_1$ steps forward with probability $p$, $k_2$ steps backward with probability $q$, and he doesn't move with probability $1-p-q$. We assume $k_1,k_2>0$.

So, in each period, the random walk takes value of $-k_1$ w/p $p$, $k_2$ w/p $q$ and $0$ w/p $1-p-q$.

At the end of each period, if he's still alive, he gets paid \$1 as a reward. The future payoff is discounted by $\delta$. So, if he is expected to fall off the cliff after 3 periods, his sum of expected rewards is $$1+\delta+\delta^2.$$

My question is what is the sum of the expected rewards?

There are some relevant questions and answers. Here Q1, Q2, Q3,... but because of the discount factor and the unequal step sizes, I can't figure out how to apply the identity... Any other approach or an answer?

Andeanlll
  • 69
  • 4
  • 14
  • Is there any limit to how far away from the cliff he can back up? If he starts at $-n$ and the cliff edge is at $0$, is the set of his possible positions bounded below? – saulspatz Aug 11 '20 at 17:35
  • 3
    If $T$ is the time at which he falls off, you want to find $E[1 + \delta + \dots + \delta^{T-1}] = \frac{1}{1-\delta}(1-E[\delta^T])$ using the geometric sum and linearity of expectation. So you need the moment generating function of $T$. As I recall this can be computed by martingale techniques though I don't know the answer off the top of my head. – Nate Eldredge Aug 11 '20 at 17:46
  • @saulspatz, Thank you for your comment. Yes. he's initial position is $n>0$. Forward in the question means a step toward the cliff and backward means a step away from the cliff. I wasn't clear about that when asking the question. Sorry about that. – Andeanlll Aug 12 '20 at 00:03
  • The money won't matter if he's guaranteed to fall off the cliff.... Jokes aside, are $k_1, k_2$ integers? – Varun Vejalla Aug 19 '20 at 03:04
  • @VarunVejalla Haha, yes. And we may assume $k$'s are integers. – Andeanlll Aug 19 '20 at 08:15
  • If $f(n) = E[\delta^T|Location=n]$ (with $f(n)=1$ if $n\leq 0$) then for $n>0$ we have $f(n) = \delta(pf(n+k_1)+qf(n-k_2))$, so this becomes a linear difference equation, you can (in principle) find the roots of $r^{k_2} = \delta p r^{k_1+k_2} + \delta q$. – Michael Aug 22 '20 at 06:16
  • @Michael Does the linear difference equation approach still work even if $k_1$ and $k_2$ are not integers? – Andeanlll Aug 29 '20 at 15:56
  • The same linear difference equation holds (replacing $n$ with $x$) and there are solutions of the type $ce^{rx}$, but now the boundary condition $f(x)=1$ for $x\leq 0$ seems more difficult to enforce, possibly requiring (infinitely?) many solutions for different $r$ values. – Michael Aug 29 '20 at 17:17

0 Answers0