Grid Search for Hyperparameter Tuning in Python

Hyperparameter tuning is the process of finding the configuration values for a machine learning model that are not learned from data — things like the depth of a decision tree, the regularization strength of a logistic regression, or the number of neurons in a neural network. Grid search is the most straightforward approach: define a discrete set of values for each hyperparameter, try every combination, and keep the best one.

This page covers:

• What hyperparameters are and why they matter
• How GridSearchCV exhaustively searches a parameter grid
• How to read cv_results_ and understand what was tested
• Using n_jobs=-1 to parallelize the search
• RandomizedSearchCV as a faster alternative for large grids
• Combining grid search with a Pipeline to avoid data leakage
• When to use grid search versus faster alternatives

All examples use scikit-learn's built-in datasets, so you can run them immediately.

What Are Hyperparameters?

Every machine learning model has two kinds of parameters:

• Model parameters are learned automatically during training (e.g., the weights in a neural network, the split thresholds in a decision tree).
• Hyperparameters are set by you before training begins. They control the learning process itself.

Choosing the wrong hyperparameters can leave a capable model severely under-performing. For example, a decision tree with no depth limit will overfit training data; one with a depth limit of 1 will underfit. The correct limit depends on your data, and grid search finds it systematically instead of by guesswork.

How GridSearchCV Works

GridSearchCV from scikit-learn combines two ideas:

1. Grid enumeration — it generates every combination of hyperparameter values you specify.
2. Cross-validation — for each combination it performs k-fold cross-validation (see Cross-Validation in Python) and records the mean score.

After the search, GridSearchCV stores the best combination and automatically re-fits a model with those settings on the full training set.

The number of fits is (combinations) × (cv folds). A grid with 3 × 3 × 3 = 27 combinations and cv=5 runs 135 fits — manageable for fast models, expensive for slow ones.

Basic GridSearchCV Example

The example below tunes a Decision Tree Classifier on the Iris dataset.

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

# Model with no hyperparameters set yet
model = DecisionTreeClassifier(random_state=42)

# Define the grid of values to try
param_grid = {
    'max_depth': [2, 3, 5, None],
    'min_samples_split': [2, 5, 10],
    'criterion': ['gini', 'entropy'],
}

# cv=5 means 5-fold cross-validation for each combination
grid_search = GridSearchCV(
    estimator=model,
    param_grid=param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1,       # use all CPU cores
    verbose=0,
)

grid_search.fit(X, y)

print("Best parameters:", grid_search.best_params_)
print("Best CV accuracy: {:.3f}".format(grid_search.best_score_))

What each argument does:

Typical output:

Best parameters: {'criterion': 'gini', 'max_depth': 3, 'min_samples_split': 2}
Best CV accuracy: 0.967

Reading cv_results_

After fitting, grid_search.cv_results_ is a dict of arrays — one entry per tested combination. The most useful keys are:

import pandas as pd

results = pd.DataFrame(grid_search.cv_results_)

# Show top 5 combinations by mean test score
cols = ['param_max_depth', 'param_min_samples_split', 'param_criterion',
        'mean_test_score', 'std_test_score', 'rank_test_score']
print(results[cols].sort_values('rank_test_score').head(5).to_string(index=False))

Key columns:

• mean_test_score — the average CV score across all folds for that combination.
• std_test_score — standard deviation; a high value means the score is unstable across folds.
• rank_test_score — rank 1 is the winner.

Scoring Options

By default GridSearchCV optimizes the estimator's default metric. You can specify any built-in scorer or a custom one:

# Common scoring strings
scoring_options = [
    'accuracy',       # classification
    'f1_weighted',    # F1 for multi-class
    'roc_auc',        # binary classification
    'neg_mean_squared_error',  # regression (note: negative so higher = better)
    'r2',             # regression
]

# Evaluate multiple metrics at once (refit on the one you care most about)
grid_search = GridSearchCV(
    estimator=DecisionTreeClassifier(random_state=42),
    param_grid={'max_depth': [2, 3, 5]},
    cv=5,
    scoring={'acc': 'accuracy', 'f1': 'f1_weighted'},
    refit='acc',       # use accuracy to pick the best model
    n_jobs=-1,
)

Using a Pipeline to Prevent Data Leakage

When your preprocessing depends on the training data (scaling, imputation, feature selection), you must fit the preprocessor only on training folds — never on the full dataset before splitting. A Pipeline handles this automatically and works seamlessly with GridSearchCV.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV

X, y = load_breast_cancer(return_X_y=True)

# Build a pipeline: scale then classify
pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('svc', SVC()),
])

# Reference pipeline steps with double-underscore: step__param
param_grid = {
    'svc__C': [0.1, 1, 10],
    'svc__kernel': ['linear', 'rbf'],
    'svc__gamma': ['scale', 'auto'],
}

grid_search = GridSearchCV(pipe, param_grid, cv=5, scoring='accuracy', n_jobs=-1)
grid_search.fit(X, y)

print("Best params:", grid_search.best_params_)
print("Best CV accuracy: {:.3f}".format(grid_search.best_score_))

The double-underscore syntax (svc__C) is the key: it tells scikit-learn to pass C to the svc step inside the pipeline. Without a pipeline, scaling the full dataset before cross-validation would leak information from the test fold into the scaler, giving an over-optimistic score.

RandomizedSearchCV: Faster for Large Grids

Exhaustive grid search becomes impractical when each hyperparameter has many candidate values. RandomizedSearchCV samples a fixed number of random combinations instead of testing all of them:

from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from scipy.stats import randint

X, y = load_iris(return_X_y=True)

# Use distributions instead of discrete lists
param_dist = {
    'n_estimators': randint(50, 500),   # random integer in [50, 500)
    'max_depth': [3, 5, 10, None],
    'min_samples_split': randint(2, 20),
    'max_features': ['sqrt', 'log2'],
}

rand_search = RandomizedSearchCV(
    estimator=RandomForestClassifier(random_state=42),
    param_distributions=param_dist,
    n_iter=30,        # try 30 random combinations instead of all
    cv=5,
    scoring='accuracy',
    n_jobs=-1,
    random_state=42,
)

rand_search.fit(X, y)

print("Best params:", rand_search.best_params_)
print("Best CV accuracy: {:.3f}".format(rand_search.best_score_))

GridSearchCV vs RandomizedSearchCV:

For large grids, RandomizedSearchCV with n_iter=50–100 frequently finds a near-optimal solution in a fraction of the compute time.

Making Predictions with the Best Model

After fitting, GridSearchCV acts like a regular estimator. Its best_estimator_ attribute holds the re-fitted model:

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

param_grid = {'max_depth': [2, 3, 5], 'criterion': ['gini', 'entropy']}
grid_search = GridSearchCV(
    DecisionTreeClassifier(random_state=42),
    param_grid, cv=5, n_jobs=-1
)
grid_search.fit(X_train, y_train)

# Evaluate on the held-out test set
test_score = grid_search.score(X_test, y_test)
print("Test accuracy: {:.3f}".format(test_score))

# Access the best model directly
best_model = grid_search.best_estimator_
predictions = best_model.predict(X_test[:5])
print("Predictions for first 5 test samples:", predictions)

Important: always keep a separate held-out test set that is never seen during the grid search. The cross-validation score inside GridSearchCV estimates generalization, but the true final evaluation should be on data the search never touched.

When to Use Grid Search

Grid search is a strong default choice when:

• You have a small or medium-sized model that trains quickly (seconds to a few minutes per fit).
• You know roughly which hyperparameters matter most and have a sensible set of candidate values.
• Reproducibility and exhaustiveness are important.

Consider alternatives when:

• The grid is large (many parameters with many values) — use RandomizedSearchCV first to narrow the space, then refine with GridSearchCV.
• Training is expensive (deep learning, large ensembles) — Bayesian optimization libraries like scikit-optimize or Optuna make smarter choices than random sampling.
• You want automatic stopping — Halving strategies (HalvingGridSearchCV) cut low-performing candidates early and require fewer total fits.

Practical Tips

• Start coarse. Use a small grid with values spread across orders of magnitude (e.g., C: [0.01, 0.1, 1, 10, 100]). Once you find a promising region, refine with a finer grid.
• Watch standard deviations. If std_test_score is large, the model is sensitive to the particular data split. Consider increasing cv or collecting more data.
• Set n_jobs=-1 to use all CPU cores — it costs nothing and often gives a 4–8× speedup on a modern machine.
• Pair with a Pipeline. Always wrap preprocessing and the model together in a Pipeline before passing to GridSearchCV. This is the single most important practice for getting reliable scores.
• Use stratified folds for classification. GridSearchCV uses StratifiedKFold automatically for classifiers, which preserves class proportions across folds.

Related Topics

• Cross-Validation in Python — understand the k-fold evaluation that powers GridSearchCV
• Decision Tree Classifier — a common model to tune with grid search
• Logistic Regression — another fast model well-suited to grid search
• K-Nearest Neighbors — n_neighbors and metric are classic grid search targets
• Linear Regression — hyperparameter tuning for regularized regression variants