Stacking and Blending

One-Line Summary: Training a meta-learner on base model predictions -- combining diverse model families for competition-winning performance.

Prerequisites: Cross-validation, overfitting, gradient boosting, random forests, linear models, regularization.

What Is Stacking?

Imagine you are deciding where to eat dinner. You ask a food critic, a health-conscious friend, and a budget-minded colleague for recommendations. Each brings a different perspective. Rather than simply going with the majority, you weigh their recommendations based on your experience of when each person tends to be right -- the critic for ambiance, the friend for healthy options, the colleague for value. You have learned a personal meta-strategy for combining their advice.

Stacking, formally known as stacked generalization and introduced by David Wolpert in 1992, applies this same principle to machine learning. It trains a collection of diverse base learners (level-0 models), then feeds their predictions into a meta-learner (level-1 model) that learns the optimal way to combine them. Unlike simple averaging or voting, the meta-learner can discover that certain base models are more reliable in certain regions of the input space and weight them accordingly.

How It Works

Stacking Architecture

The stacking framework consists of two levels:

Level 0 (Base Learners): A set of $K$ diverse models ${h_{1}, h_{2}, \dots, h_{K}}$ trained on the original features $X$ . These typically span different model families -- for example, a Random Forest, a gradient boosting model, a logistic regression, a k-nearest neighbors classifier, and a neural network.

Level 1 (Meta-Learner): A model $g$ that takes the predictions of the base learners as input features and produces the final prediction:

$\overset{y}{^}_{stack} (x) = g (h_{1} (x), h_{2} (x), \dots, h_{K} (x))$

The meta-learner is often a simple model -- logistic regression or ridge regression -- to avoid overfitting the relatively low-dimensional meta-features.

Cross-Validated Stacking (Preventing Leakage)

The critical challenge in stacking is generating unbiased meta-features. If base learners are trained on the full training set and their predictions on the same training set are used to train the meta-learner, the meta-features will be overfit (base models predict their own training data too well), and the meta-learner will learn to trust them excessively.

The solution is $K$ -fold cross-validated stacking:

Split training data into $K$ folds: $D_{1}, D_{2}, \dots, D_{K}$ .
For each fold $k = 1, \dots, K$ :
- Train each base learner $h_{m}$ on $D ∖ D_{k}$ (all folds except fold $k$ ).
- Generate predictions $\overset{y}{^}_{m}^{(k)}$ for fold $k$ using the model trained without fold $k$ .
Concatenate out-of-fold predictions to form the meta-feature matrix:

$Z = \overset{y}{^}_{1}^{(1)} \overset{y}{^}_{1}^{(2)} ⋮ \overset{y}{^}_{1}^{(K)} \overset{y}{^}_{2}^{(1)} \overset{y}{^}_{2}^{(2)} \overset{y}{^}_{2}^{(K)} \dots \dots \dots \overset{y}{^}_{K}^{(1)} \overset{y}{^}_{K}^{(2)} ⋮ \overset{y}{^}_{K}^{(K)}$

Each row corresponds to a training example, and each column corresponds to a base learner's out-of-fold prediction.

Train the meta-learner $g$ on $(Z, y)$ .
For test predictions: retrain each base learner on the full training set, generate predictions on the test set, and pass them through the meta-learner.

This procedure ensures the meta-features are generated by models that never saw the corresponding training examples, preventing leakage.

Blending (Holdout-Based Stacking)

Blending is a simplified variant of stacking that avoids the complexity of cross-validated meta-feature generation:

Split training data into a training portion (e.g., 70%) and a blending holdout (e.g., 30%).
Train all base learners on the training portion.
Generate predictions on the blending holdout.
Train the meta-learner on these holdout predictions.

Blending is simpler to implement and less prone to subtle leakage bugs, but wastes data -- base learners see less training data, and the meta-learner trains on a smaller set. It is often preferred in time-sensitive competition settings where code simplicity matters.

Multi-Level Stacking

Stacking can be extended to multiple levels:

Level 0: Diverse base learners (Random Forest, XGBoost, LightGBM, neural network, SVM, etc.)
Level 1: Meta-learners trained on level-0 predictions (possibly multiple meta-learners)
Level 2: A final meta-meta-learner combining level-1 outputs

In practice, gains diminish rapidly beyond two levels. Each additional level increases complexity, training time, and the risk of overfitting. Two-level stacking captures most of the benefit.

Choosing Diverse Base Learners

The effectiveness of stacking depends critically on diversity among base learners. If all base models make the same errors, the meta-learner has nothing to exploit. Sources of diversity include:

Different model families: Tree-based (Random Forest, gradient boosting), linear (logistic regression, ridge), instance-based (KNN), kernel-based (SVM), neural networks.
Different hyperparameters: Multiple versions of the same algorithm with different settings (e.g., a shallow and a deep Random Forest).
Different feature subsets: Training base models on different subsets of features.
Different data representations: Using raw features for some models and engineered features for others.

A practical heuristic: include models that have low pairwise correlation in their errors. If model A fails on different examples than model B, a meta-learner can learn to route predictions to the more reliable model for each region.

The Meta-Learner

The meta-learner should typically be simple and regularized:

Linear/logistic regression with regularization (ridge, lasso): Most common choice. Lasso can perform implicit model selection by zeroing out weights for unhelpful base learners.
Gradient boosting on meta-features: Can capture non-linear interactions between base model predictions. Risk of overfitting is higher.
Simple averaging: The simplest "meta-learner." Surprisingly competitive when base learners are of similar quality.

For classification problems, using predicted probabilities rather than hard class labels as meta-features provides richer information to the meta-learner.

Why It Matters

Stacking is the technique behind virtually every winning ensemble in machine learning competitions. The Netflix Prize ($1M competition), numerous Kaggle competitions, and KDD Cup victories have all relied on stacking diverse models. Beyond competitions, stacking appears in production systems where marginal accuracy improvements have significant business value -- fraud detection, medical diagnosis, financial modeling. Wolpert's original paper showed that stacking is theoretically grounded: a meta-learner can correct the biases of individual base learners, achieving lower expected loss than any single model in the ensemble.

Key Technical Details

Probability calibration: When using predicted probabilities as meta-features for classification, ensure base learners produce well-calibrated probabilities. Platt scaling or isotonic regression can be applied before stacking.
Feature augmentation: The meta-learner can receive not only base model predictions but also the original features (or a subset). This "stacking with passthrough" allows the meta-learner to learn feature-dependent weighting of base models.
Computational cost: Cross-validated stacking requires training each base learner $K$ times (once per fold) plus once on full data. For $M$ base learners with $K$ -fold CV, this means $M \times (K + 1)$ training runs.
Diminishing returns: The first few diverse base learners provide the most improvement. Adding the 10th base learner rarely helps as much as adding the 2nd or 3rd.
Target encoding risk: When base learners include models that perform target encoding (like CatBoost), extra care is needed to prevent leakage through the stacking pipeline.

Common Misconceptions

"Stacking always beats the best individual model." On small datasets, the added complexity of stacking can lead to overfitting, producing worse results than a well-tuned single model. Stacking shines on larger datasets with sufficient diversity among base learners.
"More base learners always improve stacking." Adding redundant base learners (those that make the same errors) does not help. The meta-learner's capacity is wasted on correlated meta-features. Focus on diversity, not quantity.
"The meta-learner should be complex to capture interactions." A complex meta-learner on a small number of meta-features is a recipe for overfitting. Simple regularized linear models are the standard choice and rarely need to be replaced.
"Blending and stacking are the same thing." Blending uses a fixed holdout set for meta-feature generation, while stacking uses cross-validation. Stacking is more data-efficient; blending is simpler to implement and less prone to subtle data leakage bugs.
"Stacking is too complex for production." While stacking adds inference latency (all base models must run), the architecture is straightforward to deploy: run base models in parallel, feed outputs to the meta-learner. Many production systems at major tech companies use stacking.

Connections to Other Concepts

random-forests.md: A natural base learner in stacking ensembles due to its stability and low correlation with boosting methods. Random Forests provide diversity because they reduce variance (bagging), while boosting models reduce bias.
gradient-boosting.md: Typically the strongest individual base learner in a stacking ensemble. XGBoost, LightGBM, and CatBoost are the most common gradient boosting choices.
adaboost.md: Can serve as a base learner, though modern gradient boosting implementations have largely superseded it in stacking ensembles.
bagging-and-bootstrap.md: Bagging combines identical models on different data; stacking combines different models and learns optimal weights. The meta-learner in stacking generalizes the uniform averaging in bagging.
xgboost-lightgbm-catboost.md: The workhorses of modern stacking ensembles. Using all three as base learners exploits their different tree-growing strategies and handling of features.