Logistic Regression in Python with scikit-learn

Logistic regression is a supervised classification algorithm that estimates the probability that a sample belongs to a particular class. Despite the word "regression" in its name, it is used for classification tasks — predicting a categorical label such as spam/not-spam, disease/healthy, or click/no-click.

This chapter covers:

How logistic regression works (the sigmoid function and log-odds)
Building a binary classifier in Python with scikit-learn
Evaluating a classifier beyond raw accuracy
Handling multiclass problems
Regularization and when to adjust it
Feature scaling and why it matters
When to use logistic regression vs. other classifiers

How Logistic Regression Works

From Linear Regression to Probabilities

Linear regression predicts a continuous value. If you tried to use it for classification, the predictions could fall outside [0, 1], making them impossible to interpret as probabilities. Logistic regression fixes this by passing the linear combination through the sigmoid function:

σ(z) = 1 / (1 + e^(-z))

Where z = w₀ + w₁x₁ + w₂x₂ + … + wₙxₙ is the weighted sum of input features. The sigmoid squashes any real number into the range (0, 1), giving a valid probability estimate.

Decision Boundary

The model predicts class 1 when the probability exceeds a threshold (default 0.5), and class 0 otherwise:

ŷ = 1  if σ(z) ≥ 0.5
ŷ = 0  if σ(z) < 0.5

The threshold σ(z) = 0.5 corresponds to z = 0, which defines the decision boundary — a hyperplane in feature space that separates the two classes.

Log-Odds (Logit)

Taking the log of the odds ratio shows why the model is linear in the parameters:

log(p / (1 - p)) = w₀ + w₁x₁ + … + wₙxₙ

Each coefficient wᵢ represents the change in log-odds for a one-unit increase in feature xᵢ, holding other features constant. This makes logistic regression interpretable.

How Parameters Are Learned

Unlike linear regression, there is no closed-form solution. The model minimizes the log-loss (cross-entropy) cost function using an iterative optimizer (by default lbfgs in scikit-learn):

Loss = -1/m Σ [ yᵢ log(p̂ᵢ) + (1 - yᵢ) log(1 - p̂ᵢ) ]

Lower log-loss means the predicted probabilities are better calibrated to the true labels.

Binary Classification in Python

The following example uses the Breast Cancer Wisconsin dataset — 569 samples, 30 numeric features, binary target (malignant = 1, benign = 0). It ships with scikit-learn, so no external files are needed.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, classification_report

# 1. Load data
data = load_breast_cancer()
X, y = data.data, data.target           # X: (569, 30)  y: 0=malignant, 1=benign

# 2. Split into train (80 %) and test (20 %)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# 3. Scale features — logistic regression converges faster and more reliably
#    when features are on a similar scale
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

# 4. Train
clf = LogisticRegression(max_iter=1000, random_state=42)
clf.fit(X_train_sc, y_train)

# 5. Evaluate
y_pred = clf.predict(X_test_sc)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}")
print(classification_report(y_test, y_pred, target_names=data.target_names))

Expected output:

Accuracy: 0.974
              precision    recall  f1-score   support

   malignant       0.98      0.95      0.96        43
      benign       0.97      0.99      0.98        71

    accuracy                           0.97       114
   macro avg       0.97      0.97      0.97       114
weighted avg       0.97      0.97      0.97       114

The model achieves ~97 % accuracy on held-out data. Note that scikit-learn's LogisticRegression adds L2 regularization by default (C=1.0), which helps generalisation.

Why Feature Scaling Matters

Logistic regression uses gradient-based optimisation. Without scaling, a feature with large values (e.g., mean radius ~14) dominates the gradient updates, slowing convergence or causing the solver to fail. StandardScaler transforms each feature to zero mean and unit variance.

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Without scaling — needs more iterations, may warn about convergence
clf_raw = LogisticRegression(max_iter=200, random_state=42)
clf_raw.fit(X_train, y_train)
print(f"Unscaled accuracy : {clf_raw.score(X_test, y_test):.3f}")

# With scaling
scaler = StandardScaler()
clf_sc = LogisticRegression(max_iter=200, random_state=42)
clf_sc.fit(scaler.fit_transform(X_train), y_train)
print(f"Scaled accuracy   : {clf_sc.score(scaler.transform(X_test), y_test):.3f}")

Always fit the scaler on the training set only, then use the same fitted scaler to transform both training and test sets — this prevents data leakage.

Evaluating a Classifier

Accuracy alone can be misleading when classes are imbalanced. Use a confusion matrix and the metrics derived from it.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
import matplotlib.pyplot as plt

data = load_breast_cancer()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

clf = LogisticRegression(max_iter=1000, random_state=42)
clf.fit(X_train_sc, y_train)
y_pred = clf.predict(X_test_sc)

cm = confusion_matrix(y_test, y_pred)
disp = ConfusionMatrixDisplay(cm, display_labels=data.target_names)
disp.plot(cmap="Blues")
plt.title("Logistic Regression — Breast Cancer")
plt.tight_layout()
plt.savefig("confusion_matrix.png", dpi=150)
plt.show()

Key metrics from the confusion matrix:

Metric	Formula	Meaning
Precision	TP / (TP + FP)	Of all predicted positives, how many are truly positive
Recall (Sensitivity)	TP / (TP + FN)	Of all actual positives, how many did the model catch
F1-score	2 × (P × R) / (P + R)	Harmonic mean of precision and recall
Specificity	TN / (TN + FP)	Of all actual negatives, how many did the model correctly reject

In medical diagnosis, recall (sensitivity) is often more important than precision — a missed malignancy is worse than a false alarm.

Probability Scores and the AUC-ROC Curve

Instead of a hard prediction, you can retrieve the probability of the positive class with predict_proba():

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score

data = load_breast_cancer()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
scaler = StandardScaler()
clf = LogisticRegression(max_iter=1000, random_state=42)
clf.fit(scaler.fit_transform(X_train), y_train)

# Probability of the positive class (benign = 1)
y_proba = clf.predict_proba(scaler.transform(X_test))[:, 1]
print(f"AUC-ROC: {roc_auc_score(y_test, y_proba):.3f}")

Expected output:

AUC-ROC: 0.997

An AUC close to 1.0 means the model ranks positive samples above negatives almost perfectly. See the AUC-ROC Curve chapter for how to plot and interpret the full curve.

Multiclass Classification

When the target has more than two classes, scikit-learn extends logistic regression automatically. From scikit-learn 1.5 onward, the lbfgs solver always uses the multinomial (softmax) approach, which trains a single model with a softmax output layer and minimises cross-entropy over all classes jointly. This is generally more accurate than the older One-vs-Rest (OvR) strategy, which trained a separate binary classifier per class.

The Iris dataset has three flower species — a natural multiclass example:

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score

data = load_iris()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

# From scikit-learn 1.5+, multinomial softmax is the default for lbfgs
clf = LogisticRegression(solver="lbfgs", max_iter=1000, random_state=42)
clf.fit(X_train_sc, y_train)

y_pred = clf.predict(X_test_sc)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}")

# Class probabilities for the first three test samples
print("\nClass probabilities (first 3 samples):")
for proba in clf.predict_proba(X_test_sc)[:3]:
    print([f"{p:.3f}" for p in proba])

Expected output:

Accuracy: 1.000

Class probabilities (first 3 samples):
['0.011', '0.876', '0.113']
['0.964', '0.036', '0.000']
['0.000', '0.003', '0.997']

Regularization

Regularization penalises large coefficients to prevent overfitting. scikit-learn provides two types via the penalty parameter:

Parameter	Type	Effect
`penalty='l2'` (default)	Ridge	Shrinks all coefficients toward zero; keeps all features
`penalty='l1'`	Lasso	Drives some coefficients to exactly zero; implicit feature selection
`penalty='elasticnet'`	Mix	Combines L1 and L2; requires `solver='saga'`
`penalty=None`	None	No regularization; use only if data is large and clean

The strength of regularization is controlled by C (the inverse of the regularization strength — smaller C means stronger regularization):

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np

data = load_breast_cancer()
X, y = data.data, data.target

results = {}
for C in [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]:
    pipe = Pipeline([
        ("scaler", StandardScaler()),
        ("clf", LogisticRegression(C=C, max_iter=1000, random_state=42)),
    ])
    scores = cross_val_score(pipe, X, y, cv=5, scoring="accuracy")
    results[C] = scores.mean()
    print(f"C={C:7.3f}  CV accuracy: {scores.mean():.4f} ± {scores.std():.4f}")

This uses cross-validation to find the C value that generalises best. For a systematic search over multiple hyperparameters, see Grid Search.

Using a Pipeline

A Pipeline chains preprocessing and the model into a single object. This prevents accidental data leakage and makes cross-validation and deployment simpler:

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score

data = load_breast_cancer()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression(C=1.0, max_iter=1000, random_state=42)),
])

pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}")

# Predict probabilities on a new sample (raw, unscaled)
new_sample = X_test[:1]  # first test sample
print(f"Predicted class   : {pipe.predict(new_sample)[0]}")
print(f"Class probability : {pipe.predict_proba(new_sample)[0]}")

Expected output:

Accuracy: 0.974
Predicted class   : 1
Class probability : [0.11359025 0.88640975]

The pipeline handles scaling internally — you call predict() with raw feature values.

Inspecting Coefficients

The trained coefficients reveal which features push the prediction toward each class. Larger absolute values mean stronger influence:

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
import numpy as np

data = load_breast_cancer()
X, y = data.data, data.target

scaler = StandardScaler()
clf = LogisticRegression(max_iter=1000, random_state=42)
clf.fit(scaler.fit_transform(X), y)

# Sort by absolute coefficient value
coefs = clf.coef_[0]          # shape (n_features,) for binary classification
sorted_idx = np.argsort(np.abs(coefs))[::-1]

print(f"{'Feature':<35} {'Coefficient':>12}")
print("-" * 48)
for i in sorted_idx[:5]:
    print(f"{data.feature_names[i]:<35} {coefs[i]:>12.4f}")

Expected output (top 5 features by absolute weight):

Feature                              Coefficient
------------------------------------------------
worst texture                            -1.3206
radius error                             -1.2893
worst radius                             -1.0266
area error                               -0.9989
worst area                               -0.9947

Negative coefficients (after scaling) push toward class 0 (malignant); positive coefficients push toward class 1 (benign).

Advantages and Limitations

When to use logistic regression

You need probability estimates, not just class labels.
The relationship between features and the log-odds is roughly linear.
You need an interpretable model — coefficients are meaningful.
As a fast baseline before trying more complex models like Decision Trees or ensemble methods.
Datasets are large (logistic regression scales well with many samples).

Limitations

Limitation	Mitigation
Assumes a linear decision boundary	Use polynomial features or switch to Decision Tree / K-Nearest Neighbors
Sensitive to feature scale	Always apply `StandardScaler` or `MinMaxScaler`
Struggles with highly correlated features	Drop or regularize with L1 (`penalty='l1'`)
Not suited for very complex feature interactions	Use ensemble methods or neural networks

Algorithm	Decision boundary	Scaling needed	Probabilistic output
Logistic Regression	Linear	Yes	Yes (calibrated)
Decision Tree	Non-linear (axis-aligned)	No	Yes (less calibrated)
K-Nearest Neighbors	Non-linear (instance-based)	Yes	Yes
Linear Regression	Linear (continuous output)	Yes	No

Key Takeaways

Logistic regression estimates a probability using the sigmoid function; the class is assigned by thresholding that probability.
Always scale features with StandardScaler before training — it speeds up convergence and improves accuracy.
Use a Pipeline to bundle scaling and the model; it prevents data leakage and simplifies deployment.
Evaluate with precision, recall, F1, and AUC-ROC rather than accuracy alone, especially with imbalanced data. See the Confusion Matrix and AUC-ROC Curve chapters.
Control overfitting with the C parameter (smaller = stronger regularization); use cross-validation or grid search to tune it.
For multiclass problems, use solver="lbfgs" (the default); scikit-learn 1.5+ always uses softmax (multinomial) which handles overlapping classes well.

How Logistic Regression Works

From Linear Regression to Probabilities

Decision Boundary

Log-Odds (Logit)

How Parameters Are Learned

Binary Classification in Python

Why Feature Scaling Matters

Evaluating a Classifier

Probability Scores and the AUC-ROC Curve

Multiclass Classification

Regularization

Using a Pipeline

Inspecting Coefficients

Advantages and Limitations

When to use logistic regression

Limitations

Logistic Regression vs. Related Classifiers

Key Takeaways