Normal Data Distribution

The normal distribution (also called the Gaussian distribution) is the single most important probability distribution in statistics and machine learning. Understanding it deeply — not just recognising its bell-curve shape, but knowing how to measure it, test for it, and work with it in Python — gives you a solid foundation for data preparation, model selection, and results interpretation.

This chapter covers:

• What the normal distribution is and why its shape matters
• The empirical rule (68–95–99.7 rule) and how to verify it
• Generating normally distributed data with NumPy
• Evaluating the probability density function (PDF) and cumulative distribution function (CDF) with SciPy
• Calculating and interpreting Z-scores
• Testing whether your data is normally distributed
• What to do when data is not normal

What Is the Normal Distribution?

A probability distribution describes how likely each value in a dataset is to appear. The normal distribution is a continuous, symmetric, bell-shaped distribution defined by two parameters:

• Mean (μ) — the center of the bell; where the peak sits
• Standard deviation (σ) — how wide or narrow the bell is; a larger σ spreads values further from the center

Because the curve is symmetric around the mean, the mean, median, and mode are all equal for a perfectly normal distribution.

The formula for the probability density function is:

f(x) = (1 / (σ√(2π))) × e^(−(x−μ)² / (2σ²))

You don't need to apply this formula by hand — SciPy's stats.norm handles it — but it is useful to know that the shape depends only on μ and σ.

The Empirical Rule (68–95–99.7)

One of the most practical properties of the normal distribution is the empirical rule, also called the 68–95–99.7 rule:

This means that in a normally distributed dataset, almost all values lie within three standard deviations of the mean. Values beyond ±3σ are genuinely rare — roughly 1 in 370 observations.

The following example simulates 10 000 values from a standard normal distribution (μ=0, σ=1) and verifies the rule:

import numpy as np

rng = np.random.default_rng(seed=42)
mu, sigma = 0, 1
data = rng.normal(loc=mu, scale=sigma, size=10000)

w1 = np.mean(np.abs(data - mu) < 1 * sigma) * 100
w2 = np.mean(np.abs(data - mu) < 2 * sigma) * 100
w3 = np.mean(np.abs(data - mu) < 3 * sigma) * 100

print(f"Within 1 std: {w1:.1f}%  (expected ~68%)")
print(f"Within 2 std: {w2:.1f}%  (expected ~95%)")
print(f"Within 3 std: {w3:.1f}%  (expected ~99.7%)")

Output:

Within 1 std: 67.8%  (expected ~68%)
Within 2 std: 95.4%  (expected ~95%)
Within 3 std: 99.7%  (expected ~99.7%)

The empirical rule is immediately useful: if a test score is more than two standard deviations above the mean, it falls in the top ~2.5% of all scores.

Generating Normally Distributed Data with NumPy

NumPy's random number generator produces samples that approximate a normal distribution. The rng.normal() method accepts the mean (loc), standard deviation (scale), and number of samples (size):

import numpy as np

rng = np.random.default_rng(seed=7)

# Simulate IQ scores: mean=100, std=15
samples = rng.normal(loc=100, scale=15, size=1000)

print(f"Sample mean:  {samples.mean():.1f}")
print(f"Sample std:   {samples.std():.1f}")
print(f"Min:          {samples.min():.1f}")
print(f"Max:          {samples.max():.1f}")
print(f"Sample size:  {len(samples)}")

Output:

Sample mean:  98.9
Sample std:   14.1
Min:          51.2
Max:          138.6
Sample size:  1000

The sample mean and standard deviation are close to — but not exactly — 100 and 15, because a finite sample has natural random variation. With 10 000 samples the estimates converge closer to the true parameters.

Why use a seed? Passing seed=7 to default_rng makes the results reproducible. Anyone running the same code gets the same numbers, which is essential for debugging and sharing results. The default_rng interface (introduced in NumPy 1.17) is preferred over the older np.random.normal() because it avoids shared global state.

Evaluating the PDF and CDF with SciPy

Probability Density Function (PDF)

The PDF tells you the relative likelihood of a value. A higher PDF value at x means x is more likely to occur. For a normal distribution, the peak is at the mean:

from scipy import stats

mu, sigma = 0, 1

x_values = [-2, -1, 0, 1, 2]
for x in x_values:
    pdf = stats.norm.pdf(x, loc=mu, scale=sigma)
    print(f"  pdf({x:2d}) = {pdf:.4f}")

Output:

  pdf(-2) = 0.0540
  pdf(-1) = 0.2420
  pdf( 0) = 0.3989
  pdf( 1) = 0.2420
  pdf( 2) = 0.0540

The PDF is symmetric: pdf(-1) equals pdf(1), and pdf(0) is the maximum (the peak of the bell curve). The PDF value itself is not a probability — it is a density. To get a probability, you integrate the PDF over a range, which is what the CDF does.

Cumulative Distribution Function (CDF)

The CDF at a value x gives the probability that a randomly drawn observation is less than or equal to x. Use stats.norm.cdf() to answer practical probability questions:

from scipy import stats

mu, sigma = 170, 10  # adult heights in cm

# Probability that a randomly chosen person is shorter than 180 cm
p_below_180 = stats.norm.cdf(180, loc=mu, scale=sigma)
print(f"P(height < 180 cm) = {p_below_180:.4f}")

# Probability of being taller than 185 cm
p_above_185 = 1 - stats.norm.cdf(185, loc=mu, scale=sigma)
print(f"P(height > 185 cm) = {p_above_185:.4f}")

# Probability of falling between 160 cm and 180 cm
p_range = stats.norm.cdf(180, loc=mu, scale=sigma) - stats.norm.cdf(160, loc=mu, scale=sigma)
print(f"P(160 < height < 180 cm) = {p_range:.4f}")

Output:

P(height < 180 cm) = 0.8413
P(height > 185 cm) = 0.0668
P(160 < height < 180 cm) = 0.6827

The third result confirms the empirical rule: the range μ±σ (160–180 cm) contains about 68% of observations.

Z-Scores: Standardising the Distribution

A Z-score (also called a standard score) measures how many standard deviations a value is from the mean:

Z = (x − μ) / σ

Z-scores let you compare values from distributions with different means and standard deviations on a common scale. A Z-score of +2.0 means a value is two standard deviations above the mean, regardless of the original units.

import numpy as np
from scipy import stats

heights = np.array([171.3, 168.7, 176.4, 171.0, 164.6, 173.6, 183.0, 179.5])

# Calculate Z-scores manually
mean = heights.mean()
std  = heights.std(ddof=1)   # ddof=1 for the sample standard deviation
z_manual = (heights - mean) / std
print("Z-scores (manual):", np.round(z_manual, 2))

# Or use scipy directly
z_scipy = stats.zscore(heights, ddof=1)
print("Z-scores (scipy): ", np.round(z_scipy, 2))

Output:

Z-scores (manual): [-0.37 -0.81  0.49 -0.42 -1.5   0.01  1.59  1.01]
Z-scores (scipy):  [-0.37 -0.81  0.49 -0.42 -1.5   0.01  1.59  1.01]

A person with a height of 183.0 cm has a Z-score of +1.59, meaning they are 1.59 standard deviations above the mean. In a normal distribution, this places them roughly in the top 6% of the population.

When Z-scores are useful

• Outlier detection: values with |Z| > 3 are almost certainly outliers in a normal distribution.
• Feature scaling: converting features to Z-scores (zero mean, unit variance) is called standardization, and is required for algorithms that are sensitive to feature magnitude, such as SVM and k-nearest neighbours. See the feature scaling chapter for how to apply this with scikit-learn's StandardScaler.
• Comparing apples and oranges: if you want to compare a student's math score (mean 70, std 10) with their reading score (mean 50, std 5), Z-scores give a fair comparison.

Testing Whether Data Is Normally Distributed

Before applying algorithms that assume normality, verify the assumption. The two most common approaches are the Shapiro-Wilk test (for samples up to ~5 000) and a quick visual inspection.

Shapiro-Wilk Test

The Shapiro-Wilk test returns a statistic W and a p-value. A p-value above 0.05 means there is not enough evidence to reject normality. A p-value at or below 0.05 indicates the data is unlikely to be normal.

import numpy as np
from scipy import stats

rng = np.random.default_rng(seed=42)

# Normally distributed sample
normal_sample = rng.normal(loc=0, scale=1, size=50)
stat_n, p_n = stats.shapiro(normal_sample)
print(f"Normal sample  — W={stat_n:.3f}, p={p_n:.3f}")
if p_n > 0.05:
    print("  => Cannot reject normality.")
else:
    print("  => Reject normality.")

# Skewed sample (exponential distribution)
skewed_sample = rng.exponential(scale=1, size=50)
stat_s, p_s = stats.shapiro(skewed_sample)
print(f"Skewed sample  — W={stat_s:.3f}, p={p_s:.3f}")
if p_s > 0.05:
    print("  => Cannot reject normality.")
else:
    print("  => Reject normality.")

Output:

Normal sample  — W=0.984, p=0.730
  => Cannot reject normality.
Skewed sample  — W=0.808, p=0.000
  => Reject normality.

The normal sample passes easily (p=0.730). The exponential sample fails decisively (p≈0).

When normality testing matters

Not every algorithm needs normally distributed features. Tree-based models (decision trees, random forests, gradient boosting) and neural networks are entirely distribution-agnostic. Normality matters most for:

• Parametric statistical tests (t-test, ANOVA, Pearson correlation) — these tests' p-values are only valid if the data is approximately normal.
• Linear discriminant analysis (LDA) — assumes each class is normally distributed.
• Gaussian naive Bayes — explicitly models each feature as a Gaussian.
• Confidence intervals and prediction intervals in linear regression — derived from normality of residuals.

When Data Is Not Normal: Log Transformation

A common remedy for right-skewed data (e.g., income, house prices, transaction amounts) is the log transformation, which compresses large values and stretches small ones. The result is often close to normal:

import numpy as np
from scipy import stats

rng = np.random.default_rng(seed=7)

# Simulate log-normally distributed incomes (a realistic pattern)
incomes = rng.lognormal(mean=10.5, sigma=0.5, size=1000)
log_incomes = np.log(incomes)

sk_before = stats.skew(incomes)
sk_after  = stats.skew(log_incomes)

stat_before, p_before = stats.shapiro(incomes[:50])
stat_after,  p_after  = stats.shapiro(log_incomes[:50])

print(f"Before transform — skewness: {sk_before:.2f},  Shapiro p={p_before:.3f}")
print(f"After log transform — skewness: {sk_after:.2f}, Shapiro p={p_after:.3f}")

Output:

Before transform — skewness: 1.44,  Shapiro p=0.000
After log transform — skewness: 0.01, Shapiro p=0.940

After the log transform, skewness drops from 1.44 to nearly zero, and the Shapiro-Wilk p-value rises from 0.000 to 0.940 — the transformed data easily passes the normality test.

Gotchas:

• Log transformation requires all values to be strictly positive. If your data contains zeros, use np.log(x + 1) (log-plus-one transform).
• If your data has a left skew (long tail to the left), try a square or reflection transform instead.
• Always apply the same transform to both the training set and any new data at inference time.

Why the Normal Distribution Matters in Machine Learning

The normal distribution appears everywhere in ML, often implicitly:

• Central Limit Theorem: the mean of a large random sample is approximately normally distributed, regardless of the original distribution. This underpins confidence intervals and hypothesis tests on model metrics.
• Residual analysis: the residuals of a well-fitted linear regression should be approximately normal. Deviations signal model mis-specification.
• Weight initialization in neural networks: schemes like Xavier and He initialization draw initial weights from normal distributions to prevent vanishing or exploding gradients.
• Gaussian processes: a family of probabilistic models that place a normal distribution over functions, used in Bayesian optimization and uncertainty quantification.
• Outlier detection: in many domains, data beyond ±3σ from the mean is flagged as an anomaly.

Understanding where the normal distribution is assumed — and when the assumption breaks down — helps you avoid silent modelling errors.

Summary

• The normal distribution is defined by its mean (μ) and standard deviation (σ); its bell-shaped curve is symmetric around the mean.
• The empirical rule states that 68%, 95%, and 99.7% of values fall within 1, 2, and 3 standard deviations of the mean.
• Use numpy.random.default_rng().normal() to generate normally distributed samples and scipy.stats.norm.pdf() / .cdf() to evaluate probabilities.
• Z-scores standardize values to a common scale; they are essential for outlier detection and feature scaling.
• Use scipy.stats.shapiro() to formally test for normality — but remember that many modern ML algorithms do not require it.
• When data is right-skewed, a log transform often makes it approximately normal.

For a broader comparison of distribution types (uniform, skewed, multimodal), see the Data Distribution chapter. For how to measure spread and center, see Standard Deviation and Mean, Median, and Mode.