7

Can an observed event in fact be of zero probability?

Of course, I know that there exist non-empty events of zero probability. Mu question is the reverse: given that we have observed an event (and we have no other information about it, just the fact that it has been observed), is it possible that the event has in fact zero probability? Or does an observation necessarily mean that the probability is strictly positive?

Example context for the question:

Suppose $x_i$ (countably many) are i.i.d on $[0,1]$, but we do not know the distribution they come from. It may be uniform on $[0,1]$, may be discrete, may be any legitimate distribution (discrete or continuous). Just imagine we have some sort of a machine that shows us one by one a list of randomly drawn numbers between $0$ and $1$. We are comparing the observed numbers one by one to some special number chosen beforehand, for example $\frac12$. Now, given that at some iteration we have observed that special number at least once, does that mean that $\frac12$ has some positive probability under that (unknown) distribution? And if the probability can be zero, can we nevertheless say that we will necessarily observe $\frac12$ again later, if we continue the experiment ad infimum?

Also, disregard the "real world limitations" such as an inability to produce truly uniformly distibuted numbers, or rounding errors or any such thing.

Peter Mortensen
  • 670
Aahz
  • 1,023
  • 7
    Under continuous probability models, it depends on how you interpret your observation. The probability that a randomly chosen person weighs $100\pi$ pounds is $0$. (We chose $\pi$ for its association with roundness.) But a real weight measurement, because of imprecisions, really should be reported as an interval, and the probability of lying in that interval is non-zero. – André Nicolas Sep 05 '14 at 08:44
  • I would add another interpretation, that is that a single exact event that is observed is probably a zero probability event, in the exact fashion it happened, due to the number of particules involved in most events... Depends on your definition of "event" and "observed" ;-), since you did not mention any "result". – Martigan Sep 05 '14 at 13:23
  • I have added some context to the question. Please see if that answers yours. – Aahz Sep 05 '14 at 13:35
  • 1
    If you don't know the probability distribution for your (i.i.d.) random variables, then you do not have a probability theory question. Period. And as Did remarked, in case the distribution should be uniform, then every possible observed value (assumed infinitely precise) has zero probability. And will continue to do so after it has been observed (this is the "independent" in i.i.d.). – Marc van Leeuwen Sep 05 '14 at 15:25
  • Almost never – James Sep 05 '14 at 16:00
  • @PeteBecker Randomly selecting i.i.d. coefficients of a Taylor series yields an analytic function (with radius of convergence $1$), almost surely, for most distributions of the coefficients. – Did Sep 05 '14 at 16:39
  • Duplicate of Zero probability and impossiblity. – epimorphic Jan 04 '16 at 17:30

3 Answers3

14

is it possible that the event has in fact zero probability?

Yes. For a situation where this always happens, assume that one observes a random number $x$ drawn from the uniform distribution on $(0,1)$. Then the probability to observe $x$ is zero.

(Proof: For every interval $I\subseteq(0,1)$ the probability to observe a number in $I$ is the length of $I$. For every positive $\varepsilon$, there are intervals $I\subseteq(0,1)$ which contain $x$ and have length less than $\varepsilon$ hence the probability to observe exactly $x$ is less than $\varepsilon$, for every positive $\varepsilon$, QED.)

Edit: In a discrete space $\{x_i\mid i\in\mathbb N\}$, the probability to observe $x_i$ is, by hypothesis, some positive $p_i$ hence the above applies only to continuous distributions (or at least, partially continuous).

Did
  • 279,727
  • 1
    I have thought of this example before writing. Here's what confuses me: the event of probability zero is not to observe some number (we see some number with probability one). The event with zero probability is predicting the number you draw. I.e. saying "the next number I draw will be $0.5$", and then drawing $0.5$. – Aahz Sep 05 '14 at 08:50
  • can it be extended to discrete spaces? – Alex Sep 05 '14 at 08:51
  • 1
    Yes: as you say in your question, "given that we have observed an event (and we have no other information about it, just the fact that it has been observed)", in the situation I describe, the event to observe exactly the value observed has zero probability. (Of course, this implicitly assumes that, in real life, one can choose uniformly a random number in (0,1), something which might be debatable). – Did Sep 05 '14 at 08:54
  • @Alex "can it be extended to discrete spaces? " No, see Edit. (The "Yes" in my previous comment refers to the OP's "Here's what confuses me" comment.) – Did Sep 05 '14 at 09:03
  • @Did: so for discrete spaces the answer to the OP's question is 'No'? – Alex Sep 05 '14 at 09:04
  • Yes, I see now. Thanks. This does make me think of some follow up questions. I'll let them boil a bit, and perhaps ask them later. – Aahz Sep 05 '14 at 09:14
  • @Did Actually, I do still have a question about your example. Suppose $x_i$ (countably many) are i.i.d on $[0,1]$, but we do not know their distribution. It may be uniform, may be discrete, may be anything else. We are comparing the observed numbers one by one to some special number chosen beforehand, say - $0.5$, as before. Now, given that we have observed that special number at least once, can we say that we will necessarily observe it again later? My intuition fails me, since if the distribution is continuous, we should $never$ observe $0.5$, not even that first time. – Aahz Sep 05 '14 at 10:00
  • 1
    @Aahz No, the fact that you observed it once doesn't mean you'll observe it again later. – yo' Sep 05 '14 at 10:46
  • If there are countably many possible outcomes, each with a positive probability, each outcome is observed infinitely many times hence yes, irrespectively of whether you already observed one of them or not, you will observe it again (hence @tohecz is wrong). This is a difference between discrete distributions and continuous distributions. – Did Sep 05 '14 at 11:28
  • @Did however, the question was on iid $[0,1]$. – yo' Sep 05 '14 at 12:02
  • @tohecz Quote: "It may be uniform, may be discrete, may be anything else." – Did Sep 05 '14 at 13:41
  • @Did I may have worded this second question poorly? I meant that that we have no idea what the distribution is. It may be only over the (say) rationals. Or it may be only on {$0.4,0.5,0.6$}. Or it may be some bell-shaped continuous distribution on the interval, so not necessarily countably many outcomes. The countably many refers to countably many realizations of the random variable. Please see my question, I have made an edit about this addition. – Aahz Sep 05 '14 at 14:49
  • @Did: I would have “proved” the statement in your first paragraph as follows: “Since the distribution is uniform, the probability of getting value $v$ is the same for every $v\in(0,1)$. Since there are infinitely many values, if this point probability is $>0$, then the total probability for getting some value (any value) in $(0,1)$ is infinite – but we know that it is 1, so the point probability cannot be $>0$.” Is that an oversimplification? – Scott - Слава Україні Sep 05 '14 at 17:56
  • @Did: Is the issue discrete vs. continuous, or is it finite set vs. infinite set? Can’t I have a uniform distribution over $\mathbb N$, where each possible outcome has probability $0$? … … P.S. I believe you should have said, “… the probability to observe $x_i$ is, by hypothesis, $p_i$, which is positive for at least one $i$; ….” – Scott - Слава Україні Sep 05 '14 at 17:57
4

Your question is not a mathematical question, since it depends on the interpretation of probability. It also depends on precisely what you mean by "given that we have observed...".

For example, subjective Bayesians think of probability theory as theory of how (idealized) rational agents should update their beliefs in light of new observations. It is part of this idealized framework that when an event is observed, the probability measure gets updated in such a way that the observed event gets probability 1. IOW, the a posteriori probability of an observed event is not only non-zero, but equal to 1. On the other hand, the a priori probability (the probability before the update) could be anything. Normally, Bayesians adopt additional rationality principles such as always assigning non-zero a priori probabilities to non-empty events in finite sample spaces. But modelling idealizations or misjudgements about the appropriate sample space might still amount to assigning a zero probability to events that are possible.

In an experiment where a sugar cube with faces labeled 1,2,3,4,5,6 is thrown exactly once onto a wet surface, Bob might choose the sample space {1,2,3,4,5,6} and Alice might choose the sample space {1,2,3,4,5,6,edge}. If the sugar cube ends up balancing on an edge, Alice could update her probabilities using Bayes theorem (if she assigned edge some small but non-zero probability). Bob, on the other hand, would've observed an event that he had effectively assigned a vanishing a priori probability. So he would need to update using some other rule than Bayes rule, since he also needs to expand his sample space.

There are many other interpretations of probability than subjective Bayesianism, and the detailed answer to your question may vary depending on which interpretation you prefer. This is why it is not a mathematical question.

EDIT: Regarding the added clarification to the original question, it is certainly possible (and typical!) to have a zero probability of $x_i = 1/2$ in the continuous case. In the discrete case, where the probability mass is concentrated on some finite or countably infinite discrete subset of [0,1], I am not sure. If, say, one has observed that $x_1 = \frac{1}{2}$, then intuitively it does seem plainly irrational to base predictions about $x_k$, $k > 1$, on a probability measure that assigns $P(x_k=\frac{1}{2}) = 0$. But I don't have any formal argument to support this...

Erik Tellgren
  • 151
  • I have made an edit to my question, with some context. Please see, if it makes it more precise. – Aahz Sep 05 '14 at 13:37
  • This seems to disqualify on principle every probability question as non mathematical. A little too inclusive, maybe? – Did Sep 05 '14 at 13:43
  • "In the discrete case", we know pretty well what happens. – Did Sep 05 '14 at 21:49
  • @Did, it does not disqualify every probability question, just those that depend on the interpretation of probability. The interpretation of probability has been subject to long-standing controversy precisely because it cannot be settled mathematically. – Erik Tellgren Sep 05 '14 at 21:49
  • One can rehash ad infinitum the well known frequentist/bayesian distinction, and use it to declare that a conclusion is impossible... or one can do mathematics. For example, to suggest that one could assign the mass zero to an observation in the discrete case would indeed be "plainly irrational"--and the formal argument for this is quite clear. – Did Sep 05 '14 at 21:55
  • @Did, if I understand the updated question correctly, it is not a matter of assigning zero probability to an observation. Simplifying the scenario somewhat, the question concerns what the probability of $x_2 = \frac{1}{2}$ (not observed) could be, after $x_1 = \tfrac{1}{2}$ has been observed, and subject to some i.i.d. assumption. – Erik Tellgren Sep 05 '14 at 22:22
  • Not the probability that $x_2=\frac12$ but that $x_n=\frac12$ for any $n\geqslant2$, and, as I already said, in the discrete case, we know pretty well that this probability is $1$. – Did Sep 05 '14 at 22:30
  • @Did, I don't follow. The posterior probability of an observed event is 1. But the probability that an unobserved $x_n$ takes the same value as the observed value of, say, $x_1$ is not necessarily 1. – Erik Tellgren Sep 05 '14 at 22:37
  • Once again, priors and posteriors are off topic here. The OP postulates an i.i.d. sequence $(x_n)$ with an unknown but definite distribution $p$. If $p$ is discrete and one observes $x_1=x$ then $p(x)\gt0$ hence $x_n=x$ will happen for infinitely many $n$s, almost surely. – Did Sep 05 '14 at 22:43
  • 1
    OK, I see what you meant has probability 1 now. What is your formal argument for "If p is discrete and one observes x1=x then p(x)>0"? Test case for formal arguments: Let Y be a continuous random variable uniformly distributed on [0,1]. For any arbitrary value, say 1/3, we may define the discrete variable Z = 1 if Y = 1/3 and Z = 0 otherwise. Now, observing Y=1/3 does not mean that P(Y=1/3) > 0. Equivalently, observing Z=1 does not mean that P(Z=1) > 0. PS. Maybe I should stop now, apologies to everyone for the extended exchange. – Erik Tellgren Sep 05 '14 at 23:11
  • Quote: "If p is discrete". – Did Sep 05 '14 at 23:24
  • 1
    p(Z) is discrete. Even forgetting about this kind of examples, it's genuinely unclear to me how a positive formal argument would go. – Erik Tellgren Sep 06 '14 at 00:05
1

If I understand your question correctly, this is impossible due to Bayes rule: $$ P(V|S) = \frac{P(S|V)P(V)}{P(S)} $$ where $V$ is the event. Clearly LHS is greater than zero (let's say $\epsilon$) because you have observed it. If its probability is 0 though, then $P(V)=0$ and RHS is 0, and $\epsilon \neq 0$.

Alex
  • 19,262
  • 1
    If I'm not mistaken the Bayes rule in general is problematic when you deal with continuous distributions, so relying on it may not be entirely correct. – Aahz Sep 05 '14 at 08:55
  • That's why I asked Did about it - this seems to apply to discrete spaces, but not continuous ones. – Alex Sep 05 '14 at 08:57