RobotGuide
Math foundations/Conditional Expectation: What the Vertical Bar Really Means

Foundations T15 / 27 · Probability and Expectation

Conditional Expectation: What the Vertical Bar Really Means

E[Y|X] is not a number; it is a random variable — «the part of Y that X can explain». The tower property and the projection property are where the legitimacy of TD, GAE, and least-squares value-function approximation all comes from.

≈ 60 min2 theorems6 proof steps4 self-test questions

One-line takeaway

The correct reading of conditional expectation: E[Y|X] is a function of X, and among all «predictions of Y that depend only on X» it is the one with minimal mean squared error. The tower property E[E[Y|X]]=E[Y] takes two lines to prove, yet it is the foundation of reinforcement learning's entire «take expectations in two steps» style.

The vertical bar in V(s) = E[G_t | S_t = s] — many people read papers for years without ever working out what operation it actually performs. The cost of not working it out is concrete: the definition of advantage in GAE, the order of taking expectations in n-step returns, the target of least-squares fitting — every one of these quietly uses this lesson's two theorems.

Why this tool exists

Start from intuition: given X, what is the best guess for Y? If X is today's temperature and Y is tomorrow's, then E[Y|X] is «the average of tomorrow's temperature over all days in the historical record whose temperature equals today's».

The first ambiguity appears immediately: E[Y|X=x] (given that X equals the specific value x) is a number; E[Y|X] (no value given, X itself random) is a function of X — a random variable. Mixing the two is the most common beginner mistake; the proofs below will show precisely how they relate.

The second ambiguity is deeper: in the continuous case P(X=x)=0 — there may not be a single day on which «today's temperature is exactly x», and the weighted-average formula becomes 0/0. The rigorous definition must bypass «averaging by value» and switch to «testing by blocks» — that is the motivation for partial averaging.

Definitions

Definition 3.1 Conditional expectation (intuitive version)

E[Y|X=x] is the average of Y under the condition «X=x» (weighted by conditional probability). E[Y|X] is the function obtained by replacing x with the random variable X — it varies with the value X takes, so it is a random variable and depends on the sample only through X.

Why define it this wayThe intuitive version is exactly right in the discrete case: when P(X=x)>0, just use the conditional-probability formula. Its problem is failure in the continuous case (P=0), hence the need for the rigorous version.

Definition 3.1 Conditional expectation (rigorous version: partial averaging)

E[Y|X] is defined as the random variable Z satisfying two conditions: (1) Z is a function of X (measurable with respect to X); (2) for every event A «decidable from X», the partial averages agree: E[Y·1_A] = E[Z·1_A].

Why define it this wayCondition (2) is the «block test»: don't look at single points; check whether Z and Y have the same average behavior on every block. Condition (1) guarantees Z does not peek at information beyond X. This definition exists and is unique (existence relies on the Radon–Nikodym theorem, not proved here; uniqueness takes two lines — see the note to Theorem 3.1).

Definition 3.2 The geometric reading as a projection

Regard all «functions of X» as a plane, and Y as a point off the plane. E[Y|X] is the orthogonal projection of Y onto that plane — the foot of the perpendicular. The residual Y − E[Y|X] is perpendicular to every vector in the plane.

Why define it this wayThis geometric picture is not a metaphor; it is the content of Theorem 3.2. It reappears verbatim in function approximation: least-squares fitting of a value function = projecting the returns onto a «parametrized family of functions», and the residual-orthogonality condition is the normal equations.

Theorems and proofs

Theorem 3.1 The tower property

E[ E[Y|X] ] = E[Y].

Proof

In condition (2) of Definition 3.1 take A = Ω (the whole space, certainly decidable from X — it is measurable with respect to everything):

E[Y·1_Ω] = E[E[Y|X]·1_Ω] i.e. E[Y] = E[E[Y|X]]. ∎

NoteTwo lines and done — with a good definition, the theorem comes free. Uniqueness along the way: if Z₁ and Z₂ both satisfy Definition 3.1, then E[(Z₁−Z₂)·1_A]=0 for every block; taking sets like A={Z₁−Z₂>ε} yields Z₁=Z₂ almost everywhere. The general form of the tower property (condition on the fine σ-field first, then take expectation over the coarse one, and the result equals conditioning directly on the coarse one) is the same trick stacked up.

Theorem 3.2 Projection / orthogonality

For any function Z of X (with suitable integrability), E[(Y − E[Y|X])·Z] = 0. In particular, among all functions of X, E[Y|X] is the one with minimal mean squared error against Y.

Proof

Part one: E[(Y−E[Y|X])Z] = E[YZ] − E[E[Y|X]·Z]. In the second expectation, E[Y|X]·Z is still a function of X; apply condition (2) of Definition 3.1 to it (taking A to slice by the values of Z): E[E[Y|X]·Z] = E[Y·Z]. The two terms cancel to 0. ∎

Part two (optimality): for any function f(X) of X, compare its error with the projection error:

E[(Y−f(X))²] = E[(Y−Z+Z−f(X))²] = E[(Y−Z)²] + E[(Z−f(X))²] + 2E[(Y−Z)(Z−f(X))]

In the cross term, (Z−f(X)) is a function of X, so by part one E[(Y−Z)(Z−f(X))] = 0. What remains is a sum of squares ≥ E[(Y−Z)²], with equality iff f = Z (almost everywhere). ∎

NoteThis is the rigorous version of «the conditional expectation is the best predictor». It appears verbatim in value-function approximation: fit the returns with a family {V_w}, and the optimal parameters satisfy the normal equations Σ(residual)·∂V_w/∂w = 0 — exactly the sample version of «residual orthogonal to the features».

Work it by hand

X takes ±1 with equal probability, and Y = X + ε where ε~N(0,1) is independent of X. Compute by hand E[Y], E[E[Y|X]], E[(Y−E[Y|X])²], and verify that half the variance is explained.

  1. E[Y] = E[X] + E[ε] = 0 + 0 = 0 (linearity; T2 Theorem 2.1).
  2. Conditional expectation: E[Y|X] = E[X+ε|X] = X + E[ε] = X (linearity applied inside the conditional world; ε is independent of X, so its conditional expectation is the unconditional 0).
  3. Tower property: E[E[Y|X]] = E[X] = 0 = E[Y] ✓ (holds regardless of the value X takes — this is the content of Theorem 3.1).
  4. [object Object]
  5. [object Object]
  6. Explained variance = 2 − 1 = 1, exactly Var(X). The rigorous meaning of «X explains half of Y's variance» is precisely the difference between these two expectations of squares.
Answer

E[Y]=0, E[E[Y|X]]=0, residual variance=1, total variance=2. The RL counterpart: V(s) can only explain the part of the return that «the state explains»; the rest is the environment's randomness — the critic's variance lower bound is not zero, and it is not because you didn't train enough.

Run the numbers

Where the numbers come from

Y = X² + ε, X~N(0,1), ε~N(0,1) independent (400k samples). Theoretical values: E[Y|X]=X², E[Y]=E[X²]=1:

  • Tower property: E[Y] = 1.0001, E[E[Y|X]] (bin-weighted average) = 1.0001 — matching digit for digit
  • Projection: least-squares fit of Y with features [1, x]; inner products of the residual with the two features = 0.000000 / 0.000000 (orthogonal)
  • Linear projection weights w = [1.0001, 0.0016]; theoretical optimum [1, 0] (normal equations: E[XY]=E[X³]+E[Xε]=0)
  • Bin spot-check: in the bin x∈[2.0, 2.5], E[Y|bin] = 4.89 while the bin's mean of x² ≈ 5.0 — the conditional expectation tracks x², not x

Where this shows up

The object hereThe counterpart in RLUsed in
E[Y|X] is a function of XV(s)=E[G_t|S_t=s] is a map from states to numbers, not «an independent number per state»used in →
Tower propertyGAE's λ-return splits the expectation into two steps; the order of expectations in n-step returnsused in →
Projection + orthogonalityThe normal equations of least-squares value-function approximation; residual orthogonal to featuresused in →
Best predictionThe critic's MSE lower bound is set by environment noise, not zeroused in →

Boundaries: where this article stops

Where this lesson stops: conditional expectation over general σ-fields (conditioning on «information» rather than «variables»), existence of regular conditional distributions, and version issues of conditional distributions — none covered. In the continuous case we assume throughout that the conditional expectation exists via densities — true in every routine RL setting.

Self-test: 4 questions

All hand-computation and «find the flaw in this proof» questions — nothing to memorize. Stuck? Go back to the section.

01What is the difference between E[Y|X] and E[Y|X=x]?

E[Y|X=x] is a number (the conditional average with X fixed at x); E[Y|X] is a random variable (a function of X, fluctuating with X). Theorem 3.1 says the expectation of the former (weighted over x) equals E[Y] — that is the tower property. Writing E[E[Y|X=x]] is a notation error: before averaging over x you must first restore x to a random variable.

02Why is the proof of the tower property so short? What «should» it be proving?

Because condition (2) of Definition 3.1 at A=Ω is directly the conclusion. What it should prove is that the Z prescribed by Definition 3.1 exists and is unique (existence needs Radon–Nikodym, beyond our boundary). This is exactly where a derivation course ends and a principles course begins.

03Use the «best prediction» theorem to explain: why is the critic's loss MSE rather than absolute error?

The conditional expectation E[Y|X] is precisely the optimal predictor under the MSE criterion (Theorem 3.2, part two). Under absolute error the optimal predictor becomes the conditional median — also a legitimate statistic, but the definition «V(s)=E[G|s]» corresponds to MSE. Which one to pick depends on whether you want the mean or the median, not on which is «better».

04Residual orthogonal to features — what equation is the sample version? Write the normal equations for linear features.

The least-squares normal equations: A^T(Y−Aw)=0, i.e. Σ_i (y_i − w·φ_i)φ_i = 0. Theorem 3.2's E[(Y−E[Y|X])Z]=0 is exactly this at the sample-mean level. The residual no longer contains any information about the features — the projection has «used it up».

Up next

The full series · 27 articles

T1The Matrix Is a Ledger for Rotation≈ 25 minT2From Slope to Gradient: The First Line of Every Robot Learning Algorithm≈ 25 minT3Vectors and forces: the physical reading of the dot productAbout 20 minT4Two Rulers and a Better Rulerabout 25 minT5The Ledger of One Motor Revolution: Radians, Speed, and Powerabout 20 minT6Complex multiplication is rotation: the one-dimensional version of quaternionsAbout 20 minT7Bases and coordinates: why robots put everything in matrices45 minT8Matrix Multiplication: Chaining Transformations, Not a Table OperationAbout 40 minT9The Jacobian: Rotate a Joint One Degree — How Far Does the End Effector Move?About 45 minT10Eigenvectors: The Directions a Matrix Dare Not Touchabout 45 minT11SVD and the Pseudoinverse: When There Is No Exact Solution, Find the Nearest Oneabout 50 minT12Taylor expansion: putting a straight line to work in place of a curveAbout 40 minT13Probability spaces: why «random» needs axioms50 minT14Expectation: A Rigorous Construction of «Average»About 55 minT15Conditional Expectation: What the Vertical Bar Really Means≈ 60 minT16Four inequalities: the quantitative answer to «how many samples are enough»45 minT17Norms and fixed points: where «it always converges» comes from55 minT18Markov Chains: When the Time Average Equals the Ensemble Averageabout 55 minT19The Mathematical Construction of the MDP: What One Axiom Is Worthabout 50 minT20Dynamic Programming: The Accounting for Two Iterations≈ 50 minT21Optimization: why constraints cannot rely on penalties aloneAbout 55 minT22Stochastic Approximation: Every Term in the Learning-Rate Formulaabout 55 minT23The Mathematics of Contact: Hertz Theoryabout 45 minT24Friction and the Friction Cone: The Geometry of Constraintsabout 40 minT25Force Closure and Grasp Metrics: ε Is a Linear ProgramAbout 50 minT26Contact Kinematics: Why Rolling Is Not «Sliding Along»≈ 45 minT27Slip detection: from «I felt it» to «I computed it»About 50 min