Variance Decomposition

One-Line Summary: Variance decomposition identifies whether evaluation noise comes from model sampling, environment instability, task difficulty spread, or evaluator inconsistency -- and tells you which source to fix first.

Prerequisites: sample-size-and-power-analysis.md, confidence-intervals-for-agent-metrics.md, ../01-foundations-of-agent-evaluation/what-is-agent-evaluation.md

What Is Variance Decomposition?

Imagine you are trying to measure how fast a runner completes a mile, but your measurements are noisy. The noise could come from the stopwatch (measurement error), wind conditions (environment), the runner's daily form (subject variability), or track quality (context). Without knowing which source dominates, you might invest in a better stopwatch when the real problem is the wind. Variance decomposition tells you where to invest.

In agent evaluation, observed variance in performance metrics is a mixture of several sources: the randomness inherent in LLM sampling (temperature, nucleus sampling), the instability of external environments (API timeouts, web page changes), the spread of task difficulty within the evaluation suite, and the inconsistency of the evaluation method itself (especially LLM-as-judge approaches). Each source contributes to the total variance, and they interact in complex ways.

Decomposing variance is not merely an academic exercise. It directly determines the most cost-effective strategy for improving evaluation precision. If 80% of your variance comes from model sampling, setting temperature to 0 or averaging over seeds eliminates most noise for free. If evaluator inconsistency dominates, you need a better rubric or judge -- more runs will not help much. The decomposition converts a vague "our results are noisy" into an actionable diagnosis.

How It Works

The ANOVA Model

Model agent evaluation results as a linear random-effects model. Let $Y_{ij k l}$ be the outcome for model run $l$ , evaluated by judge $k$ , on task $j$ , in environment instance $i$ :

$Y_{ij k l} = μ + α_{i} + β_{j} + γ_{k} + δ_{l} + ϵ_{ij k l}$

where:

$μ$ = grand mean (true agent performance)
$α_{i} \sim N (0, σ_{env}^{2})$ = environment variance
$β_{j} \sim N (0, σ_{task}^{2})$ = task difficulty variance
$γ_{k} \sim N (0, σ_{judge}^{2})$ = evaluator variance
$δ_{l} \sim N (0, σ_{model}^{2})$ = model sampling variance
$ϵ_{ij k l} \sim N (0, σ_{resid}^{2})$ = residual/interaction variance

The total variance decomposes as:

$σ_{total}^{2} = σ_{env}^{2} + σ_{task}^{2} + σ_{model}^{2} + σ_{judge}^{2} + σ_{resid}^{2}$

Estimation via Nested ANOVA

To estimate each component, run a factorial experiment:

Select $T$ tasks from the evaluation suite.
For each task, run the agent $R$ times with different random seeds.
For each run, instantiate $E$ environment configurations.
For each trajectory, evaluate with $J$ different judge instances (or the same LLM judge with different seeds).

Compute mean squares for each factor and solve the expected mean square equations:

$\overset{σ}{^}_{model}^{2} = \frac{M S _{model} - M S _{resid}}{E \times J}$

$\overset{σ}{^}_{env}^{2} = \frac{M S _{env} - M S _{resid}}{R \times J}$

$\overset{σ}{^}_{judge}^{2} = \frac{M S _{judge} - M S _{resid}}{R \times E}$

The Intraclass Correlation Coefficient

Report each component as a fraction of total variance using the Intraclass Correlation Coefficient (ICC):

$ICC_{source} = \frac{σ ^ _{source}^{2}}{σ ^ _{total}^{2}}$

A practical summary might look like:

Source	$\overset{σ}{^}^{2}$	ICC	Action
Model sampling	0.042	38%	Fix seed or average over $k$ runs
Environment	0.008	7%	Acceptable; use containers
Task difficulty	0.035	32%	Expected; stratify analysis
Judge inconsistency	0.021	19%	Improve rubric or use ensemble
Residual	0.004	4%	--

Practical Decomposition Protocol

A minimal but informative decomposition study requires:

Select 20+ tasks spanning the difficulty range.
Run each task 10+ times with different random seeds (model variance).
Use 2-3 environment instances per task where feasible (environment variance).
Evaluate each trajectory with 3+ judge passes (evaluator variance).
Fit the random-effects model using REML (restricted maximum likelihood) estimation.
Report ICCs and the implied optimal allocation strategy.

Total cost: $20 \times 10 \times 3 \times 3 = 1, 800$ evaluations. At $\$ 1 $/ r u n, t hi s i s$ $1{,}800$ -- a one-time investment that guides all future evaluation design.

Why It Matters

Targeted noise reduction: Instead of blindly increasing sample size, fix the dominant variance source. This can reduce required $n$ by 50-80%.
Evaluation credibility: Reporting variance components alongside results lets readers assess the reliability of each dimension independently.
Cost optimization: If model sampling dominates, use temperature=0 for evaluations. If judge noise dominates, invest in better evaluation methods rather than more runs.
Fair comparison: Two agents can only be fairly compared if the evaluation's noise floor is below the effect size of interest. Decomposition reveals whether this condition is met.
Debugging pipeline: Sudden increases in a specific variance component signal infrastructure problems (environment instability) or evaluation drift (judge model updates).

Key Technical Details

REML estimation is preferred over ML for variance components because it corrects for the downward bias in ML estimates, especially in small samples.
Negative variance estimates can occur with ANOVA-style estimation. Set these to zero and redistribute using constrained REML.
For binary outcomes, use a generalized linear mixed model (GLMM) with a logit link: $logit (p_{ij k l}) = μ + α_{i} + β_{j} + γ_{k} + δ_{l}$ . Variance components are on the latent logit scale.
Crossed vs nested designs: Tasks crossed with seeds is standard. Judges may be nested within runs (each run gets a fresh judge sample) or crossed (same judge set evaluates all runs). The design affects the estimable components.
Generalizability theory (G-theory) extends classical ANOVA by computing a generalizability coefficient $E ρ^{2}$ , which estimates how well observed scores generalize to the universe of possible evaluation conditions.

Common Misconceptions

"Just run more trials to reduce noise." If 80% of variance comes from judge inconsistency, running more agent trials barely helps. You are averaging over a small noise source while the big one persists. Always diagnose before treating.
"Temperature=0 eliminates all model variance." Temperature=0 makes sampling deterministic for a given prompt, but prompt variation (e.g., from environment state differences), batching effects, and API-level non-determinism can still introduce model-side variance.
"Task difficulty variance is noise that should be reduced." Task variance is inherent and informative -- it reflects the spread of the evaluation suite. The goal is not to reduce it but to account for it properly (see stratified-evaluation-design.md).
"One decomposition study is sufficient forever." Variance structure changes as models improve, environments are updated, and evaluation methods evolve. Re-run decomposition when any major pipeline component changes.

Connections to Other Concepts

sample-size-and-power-analysis.md -- Variance decomposition informs optimal allocation of samples across tasks and runs per task.
confidence-intervals-for-agent-metrics.md -- The relevant variance for CI computation depends on which sources are averaged over.
stratified-evaluation-design.md -- Task difficulty variance motivates stratified analysis rather than simple aggregation.
meta-evaluation.md -- Evaluator variance is a key input to meta-evaluation: a high-variance evaluator is unreliable.
../08-evaluation-tooling-and-infrastructure/evaluation-reporting.md -- Variance component reporting should be standard in evaluation dashboards.
../04-trajectory-and-process-analysis/llm-as-judge.md -- Judge inconsistency is one of the primary variance sources and a direct concern for LLM-as-judge methods.