Darth Vader Rule: what is the reason for its name, and a formal proof?

Question

I often hear the term "Darth Vader Rule" when calculating the expected value using the survival function and taking the integral where it is defined.

I am not quite sure why it is called that (is it customary?) and I would also like to know a formal proof of it. I tried to look around, but I have a feeling that the name of this rule is not official and I cannot seem to find it right away.

It's when you question a proof, and the speaker finds your lack of faith to be disturbing. — Asaf Karagila, Sep 04 '14 at 20:17
Before anyone else votes to close this question, please note that web search for "darth vader rule" does show that the term is widely-used in actuarial circles. So presumably the question, which is "what is a proof of it", is perfectly clear to someone who is familiar with actuarial matters. — MJD, Sep 04 '14 at 20:22
For intuitive explanations as well as more formal proofs, see the answers to this question. — Dilip Sarwate, Sep 04 '14 at 20:33
http://statsravingmad.wordpress.com/2010/01/25/may-the-rule-be-with-you/ says the joke is that Darth Vader was a "true survivor". I still don't get it. — Nate Eldredge, Sep 04 '14 at 21:06
The authors who apparently coined the term explain: "We propose to designate this result as the Darth Vader Rule. This is not a reference to a discoverer. But the designation may capture the somewhat counter-intuitive—if not slightly unsettling and surreal—impression which the result can evoke on first encounter." I still don't get it, but there it is. — , Sep 05 '14 at 01:05
@Rahul I think what they're saying is: "We think the result looks like nonsense. Therefore we shall give it a nonsense name. If a wookie lives on endor, the proof is correct!" — zibadawa timmy, Sep 05 '14 at 01:35

Ben · Answer 1 · 2020-09-27T01:53:29.703

On the naming question:

This expectation result has been around for a long time (e.g., you can find it in old probability books by Feller), and it appears to have been designated as the "Darth Vader rule" only quite recently. The earliest reference I can find to this name in the literature is in Muldowney, Ostaszewski and Wojdows (2012), who appear to be the ones who coined the name. They give an explanation for the name in a footnote, saying that the "...designation may capture the somewhat counterintuitive—if not slightly unsettling and surreal—impression which the result can evoke on first encounter" (p. 53, Footnote 1).

Honestly, that seems like an extremely tenuous reason for the name to me, firstly because almost every mathematical theorem seems unsettling and mysterious when you are not familiar with it, and secondly because there are plenty of other movie villains that are more unsettling and surreal than Darth Vader (the "Blair Witch rule" perhaps?). So, I think the correct answer is: there is no sensible reason why the rule is called by this name --- some maths guys just thought it would be a cool name because they are Star Wars nerds.

Notwithstanding the fact that there does not appear to be any sensible logical basis for the name, that does not really matter too much in mathematics. The main purpose of naming mathematical rules is so that we have a shared language to refer to them easily, and a silly name is just as good for this as a sensible name. For that reason, I have no problem referring to the rule by that name, and I hope it catches on widely enough that it adds to the shared language of mathematics.

score 5 · Accepted Answer · answered Sep 04 '14 at 23:31

A basic proof uses Lebesgue Integration.

Let $S(x)$ be a survival function on $x\in [0,\infty]$, then $S(x)$ is a monotonically decreasing function starting at $S(0)=1$ and $\lim\limits_{x\rightarrow \infty} S(x)= 0$.

Now, lets calculate the area under the curve using a Lebesgue Sum, of $S(x)$.

$L_S:= \sum\limits_{\eta_i\in \chi_S} \Delta(\eta_i)\mu(S^{-1}(\eta_i))$

Where:

$\chi_S$ is a partition of the range of $S(x)$ into a set of intervals.
$\Delta(\eta_i)$ is the length of interval $\eta_i \in \chi_S$
$\mu(S^{-1}(\eta_i))$ is the Lebesgue measure $\mu$ (i.e., total length) of the interval on the x-axis where $S(x)\geq \inf \eta_i$

Such an integral can be hard to interpret. However, since $S(x)$ is monotonic-decreasing, we know that the set of $x$ values in each term of the summation will have a special property: $\mu(S^{-1}(\eta_i))=x_i:S(x)=\inf \eta_i$, which means we can dispense with the Lebesgue measure and just use the actual function inverse:

$L_S := \sum\limits_{\eta_i\in \chi_S} \Delta(\eta_i)S^{-1}(\eta_i)$

Now, lets take the limit of the Lebesgue sum to get a Lebesgue Integral:

$\lim\limits_{\Delta(\eta_i)\rightarrow 0} \sum\limits_{\eta_i\in\chi_S} \eta_i\mu(S^{-1}(\eta_i)) = \int_0^1 S^{-1}(z)dz$ [This can be envisioned as the limit of a series of stacked rectangles (i.e. a Riemann sum on the inverse of S)].

However, note that $dz = dS = dP$; thus, an interval on the y-axis represents a probability, and the limit of this interval represents a density, so we can re-write the integral using the fact that $\int f(x) dx = \int f^{-1}(y) dy$:

$\int_0^1 S^{-1}(z)dz = \int_0^{\infty} xdS=\int_0^{\infty} xdP = E[X]\;\;\text{ where } F_X(x)=1-S(x)$

wow, I did not think that it would be so beyond my knowledge. Thank you for your help, though. Hopefully someday I will come back here and be able to understand your proof. — hyg17, Sep 05 '14 at 07:27
@hyg17 sorry, I didn't intend to confuse...was trying to satisfy the "formal" part of your request for a proof. I am actually an applied math person myself, so the way I actually view this is simply as a regular integral, except you treat the Y axis as the domain, and the X axis as the range (i.e., value of the function). This will get you the same result: dy=dP and the horizontal "rectangles" (in a Riemann sense) will give the area under the curve (once you integrate over all values of Y) — , Sep 05 '14 at 15:20

score 4 · Answer 3 · answered Jan 26 '21 at 03:30

On a formal proof:

There are many proofs of this rule on math.SE. Duplicate requests for proof are redirected here. Intuition on this rule can be found in this post. IMO the slickest proof argues as follows:

Claim: Let $X$ be a nonnegative random variable. Then $$ E[X]=\int_0^\infty P(X>t). $$

Proof: Write $X$ as the integral of the constant $1$ from $0$ to $X$: $$ X = \int_0^X1\,dt=\int_0^\infty H(t)dt $$ where $$H(t) = \begin{cases}1&\text{if $t<X$}\\ 0&\text{otherwise}\end{cases}$$ To compute the expectation of $X$, interchange the order of expectation and integration (Fubini-Tonelli): $$ E[X] = E\left[\int_0^\infty H(t)dt\right]\stackrel{\text{Fubini}}=\int_0^\infty E[H(t)]dt$$ But for each $t>0$, $H(t)$ is a zero-one random variable, so its expectation is the probability that it equals $1$:$$ E[H(t)] = P(H(t)=1) = P(t<X) = P(X>t).$$

The same argument proves the alternative form $E[X]=\int_0^\infty P(X\ge t)$.

Darth Vader Rule: what is the reason for its name, and a formal proof?

3 Answers3

Linked