AUC - ROC Curve

The AUC-ROC curve (Area Under the Receiver Operating Characteristic Curve) is one of the most important metrics for evaluating binary classification models. It captures a model's ability to separate positive and negative examples across every possible decision threshold, giving you a much richer picture than a single accuracy score.

This chapter covers:

• What the ROC curve is and how it is built
• How to read TPR, FPR, and the confusion-matrix terms behind them
• How to compute AUC-ROC with Python's scikit-learn
• How to plot the ROC curve with matplotlib
• When AUC-ROC is the right metric and when it is not

Key Terms: TP, FP, TN, FN

Before diving into the curve itself, it is important to understand the four outcomes that a binary classifier can produce. For any prediction, the actual label is either positive (P) or negative (N), and the predicted label is either positive or negative:

These four cells are the building blocks of every evaluation metric covered in this chapter. For a deeper introduction, see the Confusion Matrix chapter.

What is the ROC Curve?

A classifier usually outputs a probability score (a number between 0 and 1) rather than a hard label. You choose a threshold — say, 0.5 — and anything above the threshold is predicted positive.

Changing the threshold shifts the balance between catching true positives and accidentally flagging true negatives:

• A very low threshold catches more positives (high recall) but also produces more false alarms.
• A very high threshold is selective (high precision) but misses real positives.

The ROC curve plots this trade-off for every possible threshold at once:

• X-axis — False Positive Rate (FPR): what fraction of actual negatives are wrongly flagged as positive.
• Y-axis — True Positive Rate (TPR), also called Recall or Sensitivity: what fraction of actual positives are correctly identified.

TPR = TP / (TP + FN) and FPR = FP / (FP + TN)

Each point on the ROC curve is one threshold setting. Together, they trace a path from (0, 0) — the most selective threshold, predicting nothing positive — to (1, 1) — the least selective threshold, predicting everything positive.

What is AUC?

The AUC (Area Under the Curve) collapses the entire ROC curve into a single number between 0 and 1:

Intuitively, AUC is the probability that the model will rank a randomly chosen positive example higher than a randomly chosen negative example.

Computing AUC-ROC with scikit-learn

The sklearn.metrics module provides roc_curve() to get the threshold-by-threshold TPR/FPR values and roc_auc_score() to get the single AUC number.

Minimal working example

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, roc_curve

# 1. Create a toy binary-classification dataset
X, y = make_classification(n_samples=500, n_features=10, random_state=42)

# 2. Split into train / test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

# 3. Train a logistic regression model
model = LogisticRegression(random_state=42)
model.fit(X_train, y_train)

# 4. Get probability scores for the positive class (column 1)
y_proba = model.predict_proba(X_test)[:, 1]

# 5. Compute AUC
auc = roc_auc_score(y_test, y_proba)
print(f"AUC-ROC: {auc:.3f}")

# 6. Get the (fpr, tpr, thresholds) arrays for plotting
fpr, tpr, thresholds = roc_curve(y_test, y_proba)
print(f"Number of threshold points: {len(thresholds)}")

The output will be close to:

AUC-ROC: 0.944
Number of threshold points: 30

The exact value varies slightly depending on your scikit-learn version, but it should be in the 0.90–0.97 range for this dataset.

Plotting the ROC curve

import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, roc_curve

X, y = make_classification(n_samples=500, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

model = LogisticRegression(random_state=42)
model.fit(X_train, y_train)

y_proba = model.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, y_proba)
fpr, tpr, _ = roc_curve(y_test, y_proba)

plt.figure(figsize=(7, 5))
plt.plot(fpr, tpr, color="steelblue", lw=2, label=f"ROC curve (AUC = {auc:.2f})")
plt.plot([0, 1], [0, 1], color="gray", linestyle="--", label="Random baseline (AUC = 0.50)")
plt.xlabel("False Positive Rate (FPR)")
plt.ylabel("True Positive Rate (TPR)")
plt.title("ROC Curve")
plt.legend(loc="lower right")
plt.tight_layout()
plt.savefig("roc_curve.png", dpi=120)
plt.show()

The diagonal dashed line represents a random classifier. The further the ROC curve bows toward the upper-left corner, the better the model.

Comparing Multiple Models

A common workflow is to train several models and compare their ROC curves on the same plot:

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, roc_curve
import matplotlib.pyplot as plt

X, y = make_classification(n_samples=500, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

models = {
    "Logistic Regression": LogisticRegression(random_state=42),
    "Decision Tree": DecisionTreeClassifier(max_depth=3, random_state=42),
}

plt.figure(figsize=(7, 5))
for name, clf in models.items():
    clf.fit(X_train, y_train)
    y_proba = clf.predict_proba(X_test)[:, 1]
    fpr, tpr, _ = roc_curve(y_test, y_proba)
    auc = roc_auc_score(y_test, y_proba)
    plt.plot(fpr, tpr, lw=2, label=f"{name} (AUC = {auc:.2f})")

plt.plot([0, 1], [0, 1], "k--", label="Random (AUC = 0.50)")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("ROC Curve Comparison")
plt.legend(loc="lower right")
plt.tight_layout()
plt.savefig("roc_comparison.png", dpi=120)
plt.show()

The model with the larger area under its curve is generally the better choice for this dataset. For a rigorous model selection workflow, combine this with cross-validation.

Choosing an Operating Point (Threshold)

The AUC summarises all thresholds, but you will eventually need to pick one threshold for production. Two common strategies:

1. Youden's J statistic

Youden's J maximises TPR − FPR, finding the single point on the ROC curve that is furthest from the diagonal:

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_curve
import numpy as np

X, y = make_classification(n_samples=500, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

model = LogisticRegression(random_state=42)
model.fit(X_train, y_train)
y_proba = model.predict_proba(X_test)[:, 1]

fpr, tpr, thresholds = roc_curve(y_test, y_proba)
j_scores = tpr - fpr
best_idx = int(np.argmax(j_scores))
best_threshold = thresholds[best_idx]

print(f"Best threshold (Youden's J): {best_threshold:.3f}")
print(f"TPR at best threshold: {tpr[best_idx]:.3f}")
print(f"FPR at best threshold: {fpr[best_idx]:.3f}")

2. Business-driven threshold

In fraud detection you may tolerate more false positives to catch more fraud (lower threshold). In medical screening you may want very few false negatives (also lower threshold). Choosing the threshold is ultimately a business decision informed by the cost of each error type.

AUC-ROC vs. Other Metrics

For severely imbalanced data — say, 1 % positives — the AUC-PR curve (precision vs. recall) is often more informative than the AUC-ROC because FPR can look small even when many negatives are incorrectly flagged.

Common Gotchas

Passing hard labels instead of probabilities. roc_auc_score needs probability scores, not 0/1 labels. Use model.predict_proba(X_test)[:, 1], not model.predict(X_test).

Forgetting to specify the positive class. By default, roc_curve and roc_auc_score treat the larger integer label as positive. Pass pos_label=1 explicitly when your labels are not 0/1 integers.

Overfitting to AUC on the training set. Always evaluate on a held-out test set or use cross-validation to get a reliable AUC estimate. The Grid Search chapter shows how to pass scoring='roc_auc' directly to GridSearchCV.

AUC ≈ 0.5 does not always mean the model is bad. It may mean the positive and negative examples genuinely overlap in feature space, or that your features are not informative for the task.

Practical Example: Disease Prediction

The following end-to-end example uses the breast cancer dataset that ships with scikit-learn:

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, roc_curve
import matplotlib.pyplot as plt

# Load dataset (569 samples, 30 features, binary labels: malignant=0 / benign=1)
data = load_breast_cancer()
X, y = data.data, data.target

# Scale features — important for logistic regression
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(
    X_scaled, y, test_size=0.2, random_state=42, stratify=y
)

classifiers = {
    "Logistic Regression": LogisticRegression(max_iter=1000, random_state=42),
    "Random Forest": RandomForestClassifier(n_estimators=100, random_state=42),
}

plt.figure(figsize=(7, 5))
for name, clf in classifiers.items():
    clf.fit(X_train, y_train)
    y_proba = clf.predict_proba(X_test)[:, 1]
    auc = roc_auc_score(y_test, y_proba)
    fpr, tpr, _ = roc_curve(y_test, y_proba)
    plt.plot(fpr, tpr, lw=2, label=f"{name} (AUC = {auc:.3f})")

plt.plot([0, 1], [0, 1], "k--", label="Random (AUC = 0.50)")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("ROC Curve — Breast Cancer Dataset")
plt.legend(loc="lower right")
plt.tight_layout()
plt.savefig("breast_cancer_roc.png", dpi=120)
plt.show()

Both classifiers should produce AUC scores above 0.98 on this dataset, confirming that the breast cancer features are highly predictive.

Related Topics

• Confusion Matrix — the TP/FP/TN/FN building blocks used throughout this chapter
• Logistic Regression — the most common model paired with AUC-ROC evaluation
• Cross-Validation — how to get reliable AUC estimates that generalize beyond a single train/test split
• Grid Search — how to tune hyperparameters with scoring='roc_auc'
• Decision Tree — another binary classifier whose probability outputs can be evaluated with AUC-ROC

Summary

AUC-ROC gives you a single, threshold-independent number that summarises your model's discriminative ability across all operating points. Use it to compare models, tune hyperparameters, and communicate classifier quality — then pick the specific threshold that matches your application's tolerance for false positives versus false negatives.