Categorical Data

Categorical data is any data that takes on a limited, fixed set of values — think "red", "blue", "green" for a color column, or "low", "medium", "high" for a severity rating. Most machine learning algorithms work with numbers, so categorical columns must be converted to a numeric representation before training.

This chapter explains the main encoding strategies, when to choose each one, and how to apply them correctly in Python using pandas and scikit-learn without leaking information from the test set into your model.

Why Encoding Matters

Feeding raw string values to a scikit-learn estimator raises a ValueError. Even when a column already contains numbers — like 1, 2, 3 representing "small", "medium", "large" — an algorithm that treats feature values as continuous numbers will infer a false relationship (e.g., "large" is three times "small"). Encoding lets you represent the true relationship accurately.

The choice of encoding depends on two questions:

1. Is there a natural order? Color has no natural order (nominal). T-shirt size has a natural order (ordinal). The right encoding preserves order when it exists and ignores it when it does not.
2. How many distinct values (cardinality) does the column have? High-cardinality columns (hundreds of unique cities, product IDs) can create thousands of dummy columns with One-Hot Encoding, which hurts both memory and model performance.

Setting Up a Sample Dataset

The examples below use a small clothing dataset so you can follow the output exactly.

import pandas as pd

data = {
    "color":  ["red", "green", "blue", "green", "red"],
    "size":   ["S", "M", "L", "S", "M"],
    "price":  [10, 20, 30, 10, 20],
    "in_stock": [True, True, False, True, False],
}

df = pd.DataFrame(data)
print(df)

Output:

   color size  price  in_stock
0    red    S     10      True
1  green    M     20      True
2   blue    L     30     False
3  green    S     10      True
4    red    M     20     False

Label Encoding

Label Encoding replaces each category with an integer. scikit-learn's LabelEncoder assigns integers alphabetically.

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()
df["color_encoded"] = le.fit_transform(df["color"])

print(df[["color", "color_encoded"]])
print("Classes:", list(le.classes_))

Output:

   color  color_encoded
0    red              2
1  green              1
2   blue              0
3  green              1
4    red              2
Classes: ['blue', 'green', 'red']

blue → 0, green → 1, red → 2 (alphabetical order).

When to use it: Label Encoding is intended for the target variable (y), not for input features. Applied to a nominal feature column, the encoded integers imply an ordering that does not exist, which misleads tree-based models and is harmful for linear models.

Reversing the encoding:

decoded = le.inverse_transform([0, 1, 2])
print(decoded)  # ['blue' 'green' 'red']

Ordinal Encoding

Ordinal Encoding is like Label Encoding but lets you specify the exact order of categories. Use it for features where order is meaningful.

from sklearn.preprocessing import OrdinalEncoder

# Define the order explicitly: S < M < L
oe = OrdinalEncoder(categories=[["S", "M", "L"]])
df["size_encoded"] = oe.fit_transform(df[["size"]])

print(df[["size", "size_encoded"]])

Output:

  size  size_encoded
0    S           0.0
1    M           1.0
2    L           2.0
3    S           0.0
4    M           1.0

The model can now correctly infer that L (2) > M (1) > S (0).

Handling unknown categories at prediction time:

# Use handle_unknown='use_encoded_value' with unknown_value=-1
oe_safe = OrdinalEncoder(
    categories=[["S", "M", "L"]],
    handle_unknown="use_encoded_value",
    unknown_value=-1,
)
oe_safe.fit(df[["size"]])
print(oe_safe.transform([["XL"]]))  # [[-1.]]

One-Hot Encoding

One-Hot Encoding creates one binary column per category. A 1 in a column means the row belongs to that category; all other columns are 0. This is the standard choice for nominal (unordered) features fed to linear models, SVMs, and neural networks.

from sklearn.preprocessing import OneHotEncoder
import numpy as np

ohe = OneHotEncoder(sparse_output=False, handle_unknown="ignore")
color_encoded = ohe.fit_transform(df[["color"]])

# Build a labelled DataFrame from the result
col_names = ohe.get_feature_names_out(["color"])
color_df = pd.DataFrame(color_encoded, columns=col_names, dtype=int)

print(color_df)

Output:

   color_blue  color_green  color_red
0           0            0          1
1           0            1          0
2           1            0          0
3           0            1          0
4           0            0          1

handle_unknown='ignore' fills unseen categories with all zeros instead of raising an error when new data arrives at prediction time.

Dropping One Column to Avoid Multicollinearity

With three categories you get three binary columns, but the third is fully predictable from the other two (blue = 1 − green − red). This dummy variable trap can cause problems for linear models. Drop one column with drop='first':

ohe_nodrop = OneHotEncoder(sparse_output=False, drop="first", handle_unknown="ignore")
reduced = ohe_nodrop.fit_transform(df[["color"]])
print(pd.DataFrame(reduced, columns=ohe_nodrop.get_feature_names_out(["color"]), dtype=int))

Output (one column dropped):

   color_green  color_red
0            0          1
1            1          0
2            0          0
3            1          0
4            0          1

Tree-based models (decision trees, random forests, gradient boosting) are immune to the dummy variable trap, so dropping a column is optional for them.

pd.get_dummies — The Quick Pandas Alternative

For exploratory work, pd.get_dummies() is the fastest way to one-hot encode a DataFrame:

dummies = pd.get_dummies(df[["color", "size"]], dtype=int)
print(dummies)

Output:

   color_blue  color_green  color_red  size_L  size_M  size_S
0           0            0          1       0       0       1
1           0            1          0       0       1       0
2           1            0          0       1       0       0
3           0            1          0       0       0       1
4           0            0          1       0       1       0

Limitation: pd.get_dummies() is not a fitted transformer. It cannot guarantee the same column set between training and test splits, and it does not support handle_unknown. For production pipelines, prefer OneHotEncoder inside a scikit-learn Pipeline.

Avoiding Data Leakage

Data leakage occurs when information from the test set influences how the training data is prepared. The result is an overly optimistic evaluation score that does not reflect real-world performance.

The correct pattern is:

1. Split the data into train and test sets first.
2. Fit any encoder on the training set only.
3. Use transform() (not fit_transform()) on the test set.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder

X = df[["color", "size"]]
y = df["price"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)

ohe = OneHotEncoder(sparse_output=False, handle_unknown="ignore")
ohe.fit(X_train)                       # fit on training data only

X_train_enc = ohe.transform(X_train)  # transform training set
X_test_enc  = ohe.transform(X_test)   # transform test set using training-fit encoder

For details on train/test splitting, see the Train/Test Split chapter.

Using a Pipeline to Combine Encoding with a Model

A scikit-learn Pipeline chains a transformer and an estimator together. This ensures the encoder is always fit only on training data, even during cross-validation.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LinearRegression
from sklearn.compose import ColumnTransformer
from sklearn.model_selection import train_test_split

categorical_cols = ["color", "size"]
numeric_cols     = ["in_stock"]

X_full = df[categorical_cols + numeric_cols]
y_full = df["price"]

X_tr, X_te, y_tr, y_te = train_test_split(X_full, y_full, test_size=0.4, random_state=42)

preprocessor = ColumnTransformer(transformers=[
    ("ohe", OneHotEncoder(handle_unknown="ignore"), categorical_cols),
    ("pass", "passthrough", numeric_cols),
])

pipe = Pipeline(steps=[
    ("preprocessor", preprocessor),
    ("model", LinearRegression()),
])

pipe.fit(X_tr, y_tr)
print("Test predictions:", pipe.predict(X_te))

The ColumnTransformer applies different preprocessing steps to different columns in one pass. The pipeline is the recommended pattern for all production-grade machine learning workflows.

Choosing the Right Encoding

Target encoding replaces each category with the mean of the target variable for that category. It handles high cardinality well but is especially prone to data leakage — always apply it with cross-validation folds or use a library implementation (e.g., category_encoders.TargetEncoder) that handles this automatically.

Related Chapters

• Scale — normalise and standardise numeric features before modelling
• Train/Test Split — split data correctly before any preprocessing step
• Linear Regression — a model that benefits from proper categorical encoding
• Cross Validation — evaluate models reliably when combined with encoding pipelines
• Confusion Matrix — measure classification model performance after encoding targets
• Pandas Tutorial — pandas fundamentals including DataFrame creation and manipulation