Data Distribution in Machine Learning

Data distribution describes how values are spread across a dataset — which values are common, which are rare, and how far they stretch from the center. Before training any machine learning model, understanding your data's distribution helps you choose the right algorithm, detect anomalies, decide whether to scale or transform features, and avoid hidden biases in predictions.

This chapter covers the most common distribution shapes, how to measure them in Python using NumPy and SciPy, and what each distribution means for your modelling decisions.

What Is Data Distribution?

A data distribution is the pattern of how frequently each value (or range of values) appears in a dataset. When you plot a distribution, you typically see a curve or histogram whose shape tells you a great deal about the underlying data.

The three properties that define any distribution are:

• Center — where most values cluster (measured by mean, median, and mode)
• Spread — how far values deviate from the center (measured by standard deviation and interquartile range)
• Shape — whether the distribution is symmetric, skewed, or flat

Understanding all three is necessary before you feed data into a model.

Common Distribution Types

Normal Distribution

The normal distribution (also called the Gaussian distribution) is the most important distribution in statistics and machine learning. Its histogram forms a symmetric bell curve: values cluster around the mean, and frequency drops off equally on both sides.

Key properties:

• Mean, median, and mode are all equal.
• Approximately 68% of values fall within one standard deviation of the mean, 95% within two, and 99.7% within three (the empirical rule).
• Defined entirely by two parameters: mean (μ) and standard deviation (σ).

Many real-world phenomena approximate a normal distribution — human heights, measurement errors, and IQ scores are classic examples. Algorithms such as linear discriminant analysis, Gaussian naive Bayes, and linear regression assume (or benefit from) normally distributed features.

import numpy as np
from scipy import stats

# Simulate 1 000 adult heights (cm) drawn from a normal distribution
rng = np.random.default_rng(seed=42)
heights = rng.normal(loc=170, scale=10, size=1000)

print(f"Mean:   {np.mean(heights):.1f} cm")
print(f"Std:    {np.std(heights):.1f} cm")

# Verify the empirical rule
within_1_std = np.sum(np.abs(heights - 170) < 10) / 1000 * 100
within_2_std = np.sum(np.abs(heights - 170) < 20) / 1000 * 100
within_3_std = np.sum(np.abs(heights - 170) < 30) / 1000 * 100

print(f"Within 1 std: {within_1_std:.1f}%  (expected ~68%)")
print(f"Within 2 std: {within_2_std:.1f}%  (expected ~95%)")
print(f"Within 3 std: {within_3_std:.1f}%  (expected ~99.7%)")

Output:

Mean:   169.7 cm
Std:    9.9 cm
Within 1 std: 68.8%  (expected ~68%)
Within 2 std: 95.7%  (expected ~95%)
Within 3 std: 99.8%  (expected ~99.7%)

Skewed Distribution

A skewed distribution is asymmetric: one tail is longer than the other. Skewness is measured by the skewness statistic — positive values indicate a right (positive) skew, negative values indicate a left (negative) skew, and values close to zero indicate rough symmetry.

Positively skewed (right-skewed): the tail extends to the right. The mean is pulled above the median by a small number of very large values. Income and house prices are typical examples — a handful of very high earners or expensive properties drag the mean well above the median.

Negatively skewed (left-skewed): the tail extends to the left. Exam scores in an easy test often show this pattern — most students score near the top, but a few score very low.

import numpy as np
from scipy import stats

# Right-skewed: a small number of very high values pull the mean up
salaries = np.array([30000, 32000, 34000, 35000, 36000, 38000,
                     40000, 42000, 45000, 55000, 70000, 120000, 200000])

print("--- Salary distribution ---")
print(f"Mean:     {np.mean(salaries):.0f}")    # pulled up by outliers
print(f"Median:   {np.median(salaries):.0f}")  # more representative center
print(f"Skewness: {stats.skew(salaries):.2f}") # positive = right skew

# Left-skewed: most values are high, a few are very low
exam_scores = np.array([40, 68, 75, 80, 82, 85, 88, 90, 91, 92, 93, 95, 98])
print("\n--- Exam score distribution ---")
print(f"Mean:     {np.mean(exam_scores):.1f}")
print(f"Median:   {np.median(exam_scores):.1f}")
print(f"Skewness: {stats.skew(exam_scores):.2f}") # negative = left skew

Output:

--- Salary distribution ---
Mean:     59769
Median:   40000
Skewness: 2.16

--- Exam score distribution ---
Mean:     82.8
Median:   88.0
Skewness: -1.77

When skewness is present, the mean is a misleading measure of center. The median is usually a better summary statistic for skewed data.

Uniform Distribution

In a uniform distribution, every value (or range of values) is equally likely. A fair six-sided die produces a uniform discrete distribution; randomly picking a float between 0 and 1 produces a continuous uniform distribution.

import numpy as np

rng = np.random.default_rng(seed=42)
# Continuous uniform distribution between 0 and 1
uniform_data = rng.uniform(low=0, high=1, size=10000)

print(f"Mean:   {np.mean(uniform_data):.3f}  (expected 0.500)")
print(f"Std:    {np.std(uniform_data):.3f}   (expected ~0.289)")
print(f"Min:    {np.min(uniform_data):.3f}")
print(f"Max:    {np.max(uniform_data):.3f}")

Output:

Mean:   0.497  (expected 0.500)
Std:    0.288   (expected ~0.289)
Min:    0.000
Max:    1.000

Uniform distributions appear in random sampling, data augmentation, and as priors in Bayesian models.

Summarising a Distribution in Python

Before modelling, always compute a quick summary of each feature. NumPy and SciPy together cover the key statistics:

import numpy as np
from scipy import stats

data = np.array([12, 15, 14, 10, 18, 14, 13, 16, 14, 12])

print("--- Distribution Summary ---")
print(f"Mean:     {np.mean(data):.1f}")
print(f"Median:   {np.median(data):.1f}")
print(f"Std:      {np.std(data, ddof=1):.2f}")   # sample std deviation
print(f"Min:      {np.min(data)}")
print(f"Max:      {np.max(data)}")
print(f"Skewness: {stats.skew(data):.3f}")        # near 0 = symmetric

Output:

--- Distribution Summary ---
Mean:     13.8
Median:   14.0
Std:      2.25
Min:      10
Max:      18
Skewness: 0.200

A skewness near 0.200 indicates the data is roughly symmetric — no strong skew in either direction.

Testing for Normality

Some algorithms explicitly assume that features are normally distributed. The Shapiro-Wilk test is the most reliable test for small samples (n < 5 000). It returns a test statistic W and a p-value. A p-value greater than 0.05 means you cannot reject the hypothesis that the data is normal.

import numpy as np
from scipy import stats

rng = np.random.default_rng(seed=42)

# Sample from a normal distribution
normal_sample = rng.normal(loc=0, scale=1, size=50)
stat, p = stats.shapiro(normal_sample)
print(f"Shapiro-Wilk: W={stat:.3f}, p={p:.3f}")
if p > 0.05:
    print("Data appears to be normally distributed.")
else:
    print("Data does not appear to be normally distributed.")

Output:

Shapiro-Wilk: W=0.984, p=0.730
Data appears to be normally distributed.

When to use it: apply the Shapiro-Wilk test when a model's assumptions explicitly require normality — for example, before using a parametric t-test or linear discriminant analysis. Many modern algorithms (gradient boosting, random forests, neural networks) are not sensitive to the distribution shape of features, so you do not always need this test.

Why Distribution Shape Matters for Modelling

Handling Skewed Data with Log Transformation

A common remedy for right-skewed data is the log transformation, which compresses large values and expands small ones, often producing a distribution closer to normal.

import numpy as np
from scipy import stats

rng = np.random.default_rng(seed=7)
# Simulate log-normally distributed incomes (a common real-world pattern)
incomes = rng.lognormal(mean=10.5, sigma=0.5, size=1000)

print(f"Before transform — skewness: {stats.skew(incomes):.2f}")
log_incomes = np.log(incomes)
print(f"After log transform — skewness: {stats.skew(log_incomes):.2f}")

Output:

Before transform — skewness: 1.44
After log transform — skewness: 0.01

After the log transform, skewness drops from 1.44 to nearly zero, making the data far more suitable for models that assume normality.

Note: log transformation requires all values to be strictly positive. Add a small constant (e.g. np.log(x + 1)) if your data contains zeros.

Visualising Data Distribution

Visualization is the fastest way to inspect a distribution. A histogram divides values into bins and shows how many observations fall in each bin. Use Matplotlib histograms to create them:

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(seed=42)
data = rng.normal(loc=170, scale=10, size=500)

plt.figure(figsize=(8, 4))
plt.hist(data, bins=30, color='steelblue', edgecolor='white')
plt.title("Height Distribution (Normal)")
plt.xlabel("Height (cm)")
plt.ylabel("Frequency")
plt.tight_layout()
plt.show()

For a quicker overview across multiple features at once, use a scatter plot to look for relationships between distributions.

Distribution and Feature Scaling

If your features have very different distributions or scales, distance-based algorithms (k-nearest neighbours, SVM, k-means clustering) will be dominated by features with the largest range. Feature scaling is the standard fix: StandardScaler normalises each feature to a mean of 0 and standard deviation of 1, and MinMaxScaler compresses values into [0, 1].

Choose the scaler based on the underlying distribution:

• Normal distribution → StandardScaler (removes mean, scales to unit variance)
• Uniform or bounded distribution → MinMaxScaler (preserves the shape)
• Heavily skewed with outliers → RobustScaler (uses median and IQR, ignoring extreme values)

Summary

• Normal distribution: symmetric bell curve; mean ≈ median; suits parametric models.
• Skewed distribution: asymmetric; mean is pulled toward the tail; log or power transforms can reduce skewness.
• Uniform distribution: all values equally likely; appears in random sampling and as uninformative priors.
• Use scipy.stats.skew() to measure asymmetry and scipy.stats.shapiro() to formally test for normality.
• Always inspect distributions before modelling — the shape influences which algorithm to choose, how to transform features, and which scaling strategy to apply.

Next, explore Normal Data Distribution for a deeper look at the Gaussian distribution and how to generate and work with it using SciPy.