Where does the randomness come in for conditional expectations $\mathbb{E}[X | \mathcal{F}]$?

Question

For continuous random variables $X, Y$ the conditional expectation $\mathbb{E}[X | Y]$ is itself a random variable. I understood this in the sense that for a realisation of $Y$ we can say

$$ \mathbb{E}[X | Y=y] = \int_{-\infty}^{\infty}xf_{X|Y}(x|y)dx = \frac{1}{f_Y(y)}\int_{-\infty}^{\infty}xf_{X,Y}(x,y)dx. $$

And so I imagine a random elementary event happening on $\Omega$ that gives a corresponding result to $Y$ and allows conditioning $X$ as above.

The signature would then be something like this

$$\mathbb{E}[X | Y]: \Omega \to \mathbb{R}, \quad \omega \mapsto \mathbb{E}[X | Y(\omega)]$$

The conditional expectation here is random because $Y$ is a random variable that gives an output to the random occurrences on the magical space $\Omega$.

However in more advanced courses and textbooks I studied on the matter the conditional expectation is often introduced via sub-$\sigma$-algebras and then defined by some characterisation like this:

Let $X \in L_1(\Omega, \mathcal{A}, \mathbb{P})$ and $Y \in L_1(\Omega, \mathcal{F}, \mathbb{P})$ where $\mathcal{F}$ is a sub-$\sigma$-algebra of $\mathcal{A}$. Then

$$ Y = \mathbb{E}[X | \mathcal{F}] \quad \iff \quad \forall F \in \mathcal{F}: \mathbb{E}[\mathbb{1}_F X] = \mathbb{E}[\mathbb{1}_F Y]. $$

These concepts supposedly coincide as $\mathbb{E}[X | Y] = \mathbb{E}[X | \sigma(Y)]$ and $\mathbb{E}[X | \mathcal{F}]$ is understood to be a random variable as well.

My concern is with that last fact. $\mathcal{F}$ is a set of subsets of $\Omega$, so I'd say it is in principle deterministic.

Where does the randomness come in now?

What happens if the experiment that underlies $\Omega$ produces a random event? I.e. how does that influence the conditional expectation $\mathbb{E}[X | \mathcal{F}]$?

My issue here is not to doubt the usefulness of that mathematical theory but it feels like the original interpretation is not coherent with the abstraction anymore.

Answer 1

I think this a legitimate question as intuition as opposed to mere manipulation of symbols is very important in mathematics in general and especially in probability theory.

That said, it may be helpful to first understand that a $\sigma$-algebra often has to be interpreted as a body of information in the following sense. A set $A$ is contained in the $\sigma$-algebra $\mathcal{F}$ if we can answer the question "Is (the outcome of "the" probability experiment) $\omega \in A$?" (this somehow also explains the definition/closure properties of a $\sigma$-algebra!).

In the same vein the conditional expectation $\mathbb{E}\left[X \,\middle\vert\, \mathcal{F}\right](\omega)$ of a random variable $X$ given the $\sigma$-algebra (or "the information") $\mathcal{F}$ is "the best" estimate for $X(\omega)$ we can give if we are given the information necessary to answer the question "Is $\omega \in A$?" for all $A \in \mathcal{F}$.

This property is connected with the formal requirements that $$ \mathbb{E}\left[X \,\middle\vert\, \mathcal{F}\right] $$ is a $\mathcal{F}$-measurable random variable, i.e. we can actually determine it by the information $\mathcal{F}$, and the projection property $$ \mathbb{E}\left[\mathbb{E}\left[X \,\middle\vert\, \mathcal{F}\right] \cdot \mathbb{1}_{A} \right] = \mathbb{E}\left[X \cdot \mathbb{1}_{A} \right] $$ for all $A \in \mathcal{F}$ or even more general (but equivalent) $$ \mathbb{E}\left[\mathbb{E}\left[X \,\middle\vert\, \mathcal{F}\right] \cdot Z \right] = \mathbb{E}\left[X \cdot Z\right] $$ for all $\mathbb{R}_+$-valued and $\mathcal{F}$-measurable random variables $Z$. The latter expresses the idea that $\mathbb{E}\left[X \,\middle\vert\, \mathcal{F}\right]$ is the "best possible estimate of $X$ given $\mathcal{F}$".

If $A \in \mathcal{F}$ with $\mathbb{P}(A) > 0$, we can write it as $$ \frac{1}{\mathbb{P}(A)} \int_{A} \, \mathbb{E}\left[X \,\middle\vert\, \mathcal{F}\right](\omega) \, \mathbb{P}(\mathrm{d}\omega) \\ = \frac{1}{\mathbb{P}(A)} \cdot \mathbb{E}\left[\mathbb{E}\left[X \,\middle\vert\, \mathcal{F}\right] \cdot \mathbb{1}_A \right] = \frac{1}{\mathbb{P}(A)} \mathbb{E}\left[X \cdot \mathbb{1}_A \right] \\ = \frac{1}{\mathbb{P}(A)} \int_{A} \, X(\omega) \, \mathbb{P}(\mathrm{d}\omega), $$ i.e. we have the same average over all $\omega \in A$.

Another useful viewpoint (which can be used to prove existence of conditional expectations too) is that for square-integrable $X \in L^2(\Omega, \mathcal{A}, \mathbb{P})$ the conditional expectation given $\mathcal{F}$ is the $\mathcal{F}$-measurable, i.e. determined by the information $\mathcal{F}$, (and square-integrable) random variable which minimizes $$ \mathbb{E}\left[(X-Z)^2\right] $$ among all $\mathcal{F}$-measurable and square-integrable random variables $Z$.