Standard Deviation in Python

Standard deviation measures how spread out values are around their mean. It is one of the most widely used statistics in data analysis and machine learning — from checking whether a dataset is normally distributed to scaling features before training a model. This chapter explains what standard deviation is, how to compute it in Python with the statistics module, NumPy, and pandas, and how it feeds into common ML workflows.

What is Standard Deviation?

Standard deviation (σ for a population, s for a sample) quantifies the average distance each data point sits from the mean. A small standard deviation means the values cluster tightly around the mean; a large one means they are spread widely.

It is the square root of the variance:

variance  = Σ(xᵢ − x̄)² / N          # population
variance  = Σ(xᵢ − x̄)² / (N − 1)    # sample (Bessel's correction)
std dev   = √variance

Population vs. sample standard deviation

The N − 1 denominator (Bessel's correction) corrects for the bias introduced when estimating from a sample. In practice, for large datasets the difference is negligible, but it matters for small samples.

Worked example by hand

data = [10, 20, 30, 40, 50]
mean = (10 + 20 + 30 + 40 + 50) / 5 = 30

Differences from mean:  -20, -10, 0, 10, 20
Squared differences:    400, 100, 0, 100, 400

Population variance = (400 + 100 + 0 + 100 + 400) / 5   = 200
Sample variance     = (400 + 100 + 0 + 100 + 400) / 4   = 250

Population std dev  = √200 ≈ 14.14
Sample std dev      = √250 ≈ 15.81

The statistics Module

Python's built-in statistics module is the simplest way to calculate standard deviation for small datasets — no third-party libraries required.

stdev() raises StatisticsError if you pass fewer than two values, because a one-element list has no meaningful spread.

NumPy std()

NumPy is the standard choice when working with arrays, matrices, or large datasets. Its np.std() function defaults to population standard deviation (ddof=0). Pass ddof=1 to get the sample standard deviation instead.

NumPy also lets you compute standard deviation along a specific axis of a 2-D array, which is useful when each row is an observation and each column is a feature:

import numpy as np

# 3 samples, 2 features
X = np.array([[1, 10],
              [2, 20],
              [3, 30]])

print(np.std(X, axis=0, ddof=1))  # std per feature: [1. 10.]
print(np.std(X, axis=1, ddof=1))  # std per sample:  [6.36 6.36 6.36] (approx)

pandas std() and describe()

When your data lives in a DataFrame, pandas provides std() directly on any column or the whole frame. By default pandas uses ddof=1 (sample std), matching R's convention.

import pandas as pd

temps = [72, 68, 75, 80, 65, 70, 78]
df = pd.DataFrame({"temperature": temps})

print(df["temperature"].std())    # 5.4116  (sample std, ddof=1)
print(df["temperature"].mean())   # 72.5714

describe() gives you a quick statistical summary including the standard deviation for every numeric column:

import pandas as pd

df = pd.DataFrame({
    "height_cm": [165, 170, 175, 160, 180],
    "weight_kg": [55,  70,  80,  50,  90],
})

print(df.describe())

The std row in the output shows the sample standard deviation for each column:

       height_cm  weight_kg
count   5.000000   5.000000
mean  170.000000  69.000000
std     7.905694  16.733201
min   160.000000  50.000000
25%   165.000000  55.000000
50%   170.000000  70.000000
75%   175.000000  80.000000
max   180.000000  90.000000

Standard Deviation in Machine Learning

Feature scaling with StandardScaler

Raw features often span very different scales (age in years vs. income in thousands). Algorithms like linear regression, SVMs, and k-nearest neighbours are sensitive to this imbalance. Standardisation (also called z-score normalisation) transforms each feature so it has mean = 0 and standard deviation = 1:

z = (x − mean) / std

scikit-learn's StandardScaler applies this automatically:

from sklearn.preprocessing import StandardScaler
import numpy as np

# 3 samples, 2 features
features = np.array([[1, 2],
                     [3, 4],
                     [5, 6]])

scaler = StandardScaler()
scaled = scaler.fit_transform(features)

print(scaled)
# [[-1.2247  -1.2247]
#  [ 0.       0.    ]
#  [ 1.2247   1.2247]]

After scaling, every column has a mean of 0 and a standard deviation of 1. You can inspect the learned parameters later:

print(scaler.mean_)   # [3. 4.]
print(scaler.scale_)  # [1.6330 1.6330]

Comparing group variability

Standard deviation helps you decide whether two groups are truly different or just noisy. Two classes can share a similar mean but differ dramatically in spread:

import statistics

scores_a = [78, 80, 82, 79, 81]   # consistent group
scores_b = [60, 100, 55, 95, 70]  # high-variance group

print(f"Group A — mean: {statistics.mean(scores_a)}, std: {statistics.stdev(scores_a):.2f}")
# Group A — mean: 80, std: 1.58

print(f"Group B — mean: {statistics.mean(scores_b)}, std: {statistics.stdev(scores_b):.2f}")
# Group B — mean: 76, std: 20.43

Group A and Group B have similar means, but Group B's standard deviation is ~13× larger, indicating far less predictable outcomes.

Anomaly detection with z-scores

A z-score tells you how many standard deviations a value sits from the mean. Values with |z| > 3 are conventional outlier candidates:

import numpy as np

values = np.array([2.0, 2.5, 3.0, 2.8, 100.0, 2.2, 3.1])

mean = np.mean(values)
std  = np.std(values, ddof=1)

z_scores = (values - mean) / std
print(z_scores.round(2))
# [-0.39 -0.38 -0.37 -0.37  2.27 -0.39 -0.36]

The value 100.0 has a z-score of 2.27, which — given the tiny dataset — already stands out strongly as a probable outlier.

Understanding model uncertainty

Standard deviation also appears when you evaluate a model over multiple cross-validation folds. A high standard deviation across folds suggests the model is unstable or that the data has high variance. See the cross-validation chapter for a full walkthrough.

Choosing the Right Tool

Common Pitfalls

• Wrong ddof: NumPy defaults to ddof=0 (population) while pandas defaults to ddof=1 (sample). Always check which you need before comparing results from the two libraries.
• Single-element lists: statistics.stdev() raises an error; np.std() returns 0.0 silently.
• Fitting the scaler on test data: always call scaler.fit_transform() on the training set and scaler.transform() (not fit_transform) on the test set. Fitting on test data leaks information and inflates performance metrics. See the train/test split chapter for details.
• Outliers distort the std: a single extreme value can inflate standard deviation dramatically, as shown in the anomaly-detection example above. Consider checking for outliers first (see the data distribution chapter).

Related Chapters

• Mean, Median, and Mode — other central-tendency measures that complement standard deviation
• Percentiles — rank-based spread measures that are robust to outliers
• Normal Data Distribution — standard deviation is the core parameter of the normal distribution
• Scale — broader feature-scaling strategies beyond standardisation
• Data Distribution — visualising spread and identifying skew