Bootstrap Aggregation (Bagging) in Python

Bootstrap aggregation — commonly called bagging — is an ensemble technique that trains multiple models on different random samples of your data and then combines their predictions. The result is a model that is less sensitive to noise and outliers, and that generalises better to unseen data than any single model trained on the full dataset.

This chapter covers:

• What bootstrap sampling is and why it helps
• The bias–variance tradeoff that bagging addresses
• How to implement BaggingClassifier and BaggingRegressor with scikit-learn
• Key hyperparameters and how to tune them
• Out-of-bag (OOB) error as a free validation estimate
• When bagging helps and when to prefer other methods

How bootstrap sampling works

The word bootstrap refers to sampling with replacement. Given a dataset of n examples, one bootstrap sample is created by drawing n examples at random, where each draw is independent and the same example can appear more than once.

import numpy as np

rng = np.random.default_rng(42)
data = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

bootstrap_sample = rng.choice(data, size=len(data), replace=True)
out_of_bag = np.setdiff1d(data, bootstrap_sample)

print("Original data:   ", data)
print("Bootstrap sample:", bootstrap_sample)
print("Out-of-bag:      ", out_of_bag)

Output:

Original data:    [ 1  2  3  4  5  6  7  8  9 10]
Bootstrap sample: [1 8 7 5 5 9 1 7 3 1]
Out-of-bag:       [ 2  4  6 10]

On average, a bootstrap sample contains about 63 % of the unique original examples; the remaining 37 % are left out. These out-of-bag (OOB) examples can be used as a built-in validation set without needing a separate hold-out split.

The bias–variance tradeoff

Every model makes predictions with some combination of bias (systematic error from wrong assumptions) and variance (sensitivity to fluctuations in the training data). Bagging targets high-variance, low-bias models — particularly deep decision trees.

A single unpruned decision tree can memorise the training set perfectly but fails badly on new data. By averaging the predictions of many trees, each trained on a slightly different bootstrap sample, the random fluctuations cancel out and variance drops — without meaningfully increasing bias.

Bagging does not help models that already have high bias (for example, shallow linear models) because averaging many biased models still gives a biased answer. For those cases, boosting methods such as Gradient Boosting are more appropriate.

Implementing BaggingClassifier

scikit-learn provides BaggingClassifier for classification tasks. The example below compares a single decision tree against a bagged ensemble on the breast cancer dataset.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import accuracy_score

# Load dataset
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Baseline: single decision tree
single_tree = DecisionTreeClassifier(random_state=42)
single_tree.fit(X_train, y_train)
single_acc = accuracy_score(y_test, single_tree.predict(X_test))

# Bagging ensemble of 50 decision trees
bagging_model = BaggingClassifier(
    estimator=DecisionTreeClassifier(),
    n_estimators=50,
    random_state=42,
)
bagging_model.fit(X_train, y_train)
bagging_acc = accuracy_score(y_test, bagging_model.predict(X_test))

print(f"Single tree accuracy:  {single_acc:.2f}")
print(f"Bagging accuracy:      {bagging_acc:.2f}")

Output:

Single tree accuracy:  0.95
Bagging accuracy:      0.96

The bagged model outperforms the single tree. On noisier or smaller datasets the margin is usually larger.

Key constructor parameters

Out-of-bag (OOB) error

Because each base learner only sees about 63 % of the training data, the remaining 37 % can be used to evaluate that learner without touching the test set. Averaging these OOB evaluations across all estimators gives the OOB score — a nearly free estimate of test accuracy.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

bagging_oob = BaggingClassifier(
    estimator=DecisionTreeClassifier(),
    n_estimators=50,
    oob_score=True,      # enable OOB evaluation
    random_state=42,
)
bagging_oob.fit(X_train, y_train)

print(f"OOB score: {bagging_oob.oob_score_:.2f}")

Output:

OOB score: 0.96

The OOB score is close to the held-out test accuracy, making it useful as a quick sanity check — especially when your dataset is too small to afford a separate validation split. For a more rigorous estimate, pair bagging with cross-validation.

BaggingRegressor

Bagging is equally useful for regression. Replace BaggingClassifier with BaggingRegressor and choose a regression base learner.

from sklearn.ensemble import BaggingRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
from sklearn.datasets import fetch_california_housing
from sklearn.metrics import mean_squared_error
import numpy as np

X, y = fetch_california_housing(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

bagging_reg = BaggingRegressor(
    estimator=DecisionTreeRegressor(),
    n_estimators=50,
    random_state=42,
)
bagging_reg.fit(X_train, y_train)

rmse = np.sqrt(mean_squared_error(y_test, bagging_reg.predict(X_test)))
print(f"BaggingRegressor RMSE: {rmse:.4f}")

Output:

BaggingRegressor RMSE: 0.5080

Evaluating with cross-validation

A single train-test split can give an overly optimistic or pessimistic picture depending on which examples land in each partition. Running cross-validation averages the result across multiple splits for a more reliable score.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

clf = BaggingClassifier(
    estimator=DecisionTreeClassifier(),
    n_estimators=50,
    random_state=42,
)

scores = cross_val_score(clf, X, y, cv=5)
print(f"CV scores: {scores.round(4)}")
print(f"Mean: {scores.mean():.4f}, Std: {scores.std():.4f}")

Output:

CV scores: [0.9123 0.9211 0.9825 0.9561 1.    ]
Mean: 0.9544, Std: 0.0339

A low standard deviation across folds means the model generalises consistently. A high standard deviation suggests the model is sensitive to which data ends up in each fold.

Bagging vs. Random Forest

Random Forest is the most popular bagging-based algorithm. It extends plain bagging by also randomly selecting a subset of features at each split decision — not just a subset of rows — which decorrelates the trees further and often gives better accuracy.

When you are bagging decision trees, RandomForestClassifier is almost always the better choice. Use BaggingClassifier when you want to bag a different base learner — for example, a KNeighborsClassifier from K-Nearest Neighbors or a logistic regression from Logistic Regression.

When to use bagging

Bagging is most effective when:

• Your base model has high variance (deep decision trees, high-degree polynomial models).
• You have enough data to make diverse bootstrap samples meaningful.
• You can afford parallel training, because each base learner trains independently and the workload can be distributed across CPU cores (n_jobs=-1).

It is less helpful when:

• The base model already has low variance (e.g., linear models with strong regularisation).
• You need a single, interpretable model — an ensemble of 50 trees is not easy to explain to a stakeholder.
• Computational cost matters more than accuracy — training 50 models is 50 times slower than training one.

For high-bias models, consider Grid Search to tune hyperparameters, or switch to a boosting method. For evaluation, always verify your model on a held-out split created with train-test split or with cross-validation.

Summary

• Bootstrap aggregation trains many base learners on random samples of the data (drawn with replacement) and averages their predictions.
• It reduces variance without significantly increasing bias, making it ideal for high-variance models such as deep decision trees.
• scikit-learn provides BaggingClassifier and BaggingRegressor; key parameters are n_estimators, max_samples, and max_features.
• Enable oob_score=True for a free generalization estimate that does not require a separate validation set.
• When bagging trees, RandomForestClassifier is usually preferable; use BaggingClassifier to bag other model types.