Pandas Tutorial for Python — Complete Beginner's Guide

pandas is the most widely used Python library for working with structured (tabular) data. It provides two core data structures — Series and DataFrame — and a rich set of tools for loading, cleaning, transforming, aggregating, and visualising data.

This chapter covers everything you need to go from zero to productive with pandas: installation, data structures, reading files, selecting and filtering data, handling missing values, grouping, merging, and basic plotting.

Installing pandas

Install pandas with pip:

pip install pandas

If you plan to follow the plotting examples later in this chapter, also install Matplotlib:

pip install matplotlib

After installation, import pandas using the conventional alias pd:

import pandas as pd

The Two Core Data Structures

Series

A Series is a one-dimensional labelled array. Each element has a value and a corresponding index label (by default, integers starting at 0).

import pandas as pd

scores = pd.Series([88.5, 92.0, 79.5, 95.0])
print(scores)

Output:

0    88.5
1    92.0
2    79.5
3    95.0
dtype: float64

You can supply custom index labels to make the data self-describing:

scores = pd.Series(
    [88.5, 92.0, 79.5, 95.0],
    index=['Alice', 'Bob', 'Carol', 'Dave']
)
print(scores)
print(scores['Bob'])  # 92.0

Output:

Alice    88.5
Bob      92.0
Carol    79.5
Dave     95.0
dtype: float64
92.0

A Series with a meaningful index is similar to a Python dictionary but supports vectorised operations and is integrated with the rest of the pandas API.

DataFrame

A DataFrame is a two-dimensional table with labelled rows and columns — think of it as a spreadsheet or a SQL table loaded into Python. Each column is internally a Series that shares the same row index.

Creating a DataFrame from a dictionary of lists

python— editable, runs on the server

Output:

    name  age  score
0  Alice   25   88.5
1    Bob   30   92.0
2  Carol   28   79.5
3   Dave   35   95.0

The dictionary keys become column names; the dictionary values become columns. The row index defaults to 0, 1, 2, 3.

Useful attributes to inspect a DataFrame right after creation:

Attribute / Method	What it returns
`df.shape`	Tuple of `(rows, columns)`
`df.dtypes`	Data type of each column
`df.columns`	Column labels
`df.index`	Row labels
`df.head(n)`	First `n` rows (default 5)
`df.tail(n)`	Last `n` rows (default 5)
`df.describe()`	Summary statistics for numeric columns
`df.info()`	Column names, types, and non-null counts

Reading Data from Files

In real projects you rarely create DataFrames by hand. pandas can read dozens of file formats.

Reading a CSV file

import pandas as pd

df = pd.read_csv('sales.csv')
print(df.head())
print(df.shape)   # e.g. (1000, 5)
print(df.dtypes)

pd.read_csv() automatically infers column types. Common options:

sep=';' — change the delimiter (default is comma)
index_col='id' — use a column as the row index
parse_dates=['date'] — parse a column as datetime
nrows=500 — only read the first 500 rows (useful for large files)

Reading an Excel file

df = pd.read_excel('report.xlsx', sheet_name='Sheet1')

Requires the optional dependency openpyxl: pip install openpyxl.

Writing data back to a file

df.to_csv('output.csv', index=False)     # CSV without the row index column
df.to_excel('output.xlsx', index=False)  # Excel

Selecting Data

Selecting columns

Select a single column to get a Series:

ages = df['age']
print(type(ages))  # <class 'pandas.core.series.Series'>

Select multiple columns to get a DataFrame:

subset = df[['name', 'score']]
print(subset)

Output:

    name  score
0  Alice   88.5
1    Bob   92.0
2  Carol   79.5
3   Dave   95.0

Selecting rows with `loc` and `iloc`

pandas provides two main row selectors:

loc — select by label (index value)
iloc — select by integer position (0-based)

import pandas as pd

data = {
    'name':  ['Alice', 'Bob', 'Carol', 'Dave'],
    'age':   [25, 30, 28, 35],
    'score': [88.5, 92.0, 79.5, 95.0],
}
df = pd.DataFrame(data)

print(df.loc[1])          # row with index label 1
print()
print(df.iloc[0])         # first row by position
print()
print(df.loc[1:2])        # rows with labels 1 and 2 (inclusive)
print()
print(df.iloc[0:2])       # rows at positions 0 and 1 (end is exclusive)

Output:

name      Bob
age        30
score    92.0
Name: 1, dtype: object

name     Alice
age         25
score     88.5
Name: 0, dtype: object

   name  age  score
1   Bob   30   92.0
2  Carol   28   79.5

    name  age  score
0  Alice   25   88.5
1    Bob   30   92.0

Note the difference: loc[1:2] is label-based and inclusive on both ends; iloc[0:2] is position-based and exclusive on the right end (like regular Python slices).

You can also select a specific cell:

print(df.loc[1, 'score'])   # 92.0  — by label
print(df.iloc[0, 2])        # 88.5  — by position

Filtering Rows

Filter rows using a boolean condition inside square brackets:

import pandas as pd

data = {
    'name':  ['Alice', 'Bob', 'Carol', 'Dave'],
    'age':   [25, 30, 28, 35],
    'score': [88.5, 92.0, 79.5, 95.0],
}
df = pd.DataFrame(data)

# Rows where age is greater than 28
print(df[df['age'] > 28])

Output:

   name  age  score
1   Bob   30   92.0
3  Dave   35   95.0

Combine multiple conditions with & (and) or | (or). Always wrap each condition in parentheses:

# Age > 25 AND score >= 90
print(df[(df['age'] > 25) & (df['score'] >= 90)])

Output:

   name  age  score
1   Bob   30   92.0
3  Dave   35   95.0

Use isin() to filter by a list of values:

print(df[df['name'].isin(['Alice', 'Carol'])])

Adding and Modifying Columns

Add a new column by assigning to it:

import pandas as pd

data = {
    'name':  ['Alice', 'Bob', 'Carol', 'Dave'],
    'age':   [25, 30, 28, 35],
    'score': [88.5, 92.0, 79.5, 95.0],
}
df = pd.DataFrame(data)

# Add a column with a calculated value
df['grade'] = df['score'].apply(lambda x: 'A' if x >= 90 else 'B')
print(df)

Output:

    name  age  score grade
0  Alice   25   88.5     B
1    Bob   30   92.0     A
2  Carol   28   79.5     B
3   Dave   35   95.0     A

apply() runs a function on every element of a column. For simple arithmetic you can use vectorised operations directly — they are faster than apply():

df['score_scaled'] = df['score'] / 100   # no apply() needed

Rename columns with rename():

df = df.rename(columns={'score': 'exam_score', 'age': 'years_old'})

Drop a column with drop():

df = df.drop(columns=['grade'])

Sorting Data

Sort a DataFrame by one or more columns using sort_values():

import pandas as pd

data = {
    'department': ['Eng', 'Eng', 'HR', 'HR', 'Eng'],
    'name':       ['Alice', 'Bob', 'Carol', 'Dave', 'Eve'],
    'salary':     [90000, 95000, 70000, 72000, 88000],
}
df = pd.DataFrame(data)

print(df.sort_values('salary', ascending=False))

Output:

  department   name  salary
1        Eng    Bob   95000
0        Eng  Alice   90000
4        Eng    Eve   88000
3         HR   Dave   72000
2         HR  Carol   70000

Sort by multiple columns — for example, department ascending, then salary descending within each department:

df_sorted = df.sort_values(['department', 'salary'], ascending=[True, False])
print(df_sorted)

Handling Missing Values

Real-world datasets almost always contain missing values. pandas represents them as NaN (Not a Number).

import pandas as pd

data = {
    'name':  ['Alice', 'Bob', 'Carol', 'Dave'],
    'age':   [25, None, 28, 35],
    'score': [88.5, 92.0, None, 95.0],
}
df = pd.DataFrame(data)

# Count missing values per column
print(df.isnull().sum())

Output:

name     0
age      1
score    1
dtype: int64

Filling missing values

Replace NaN with a fixed value or a statistic:

df_filled = df.fillna({
    'age':   df['age'].mean(),    # fill with column mean
    'score': df['score'].median() # fill with column median
})
print(df_filled)

Output:

    name        age  score
0  Alice  25.000000   88.5
1    Bob  29.333333   92.0
2  Carol  28.000000   92.0
3   Dave  35.000000   95.0

Dropping rows with missing values

df_dropped = df.dropna()
print(df_dropped)

Output:

    name   age  score
0  Alice  25.0   88.5
3   Dave  35.0   95.0

When to fill vs. when to drop: dropping rows loses data and can introduce bias if values are not missing at random. Filling (imputation) is usually preferable when missing data is sparse. For machine learning workflows, see the Scale chapter for standardising numeric features after imputation.

Grouping Data

groupby() splits the DataFrame into groups, applies a function to each group, and combines the results. This is the pandas equivalent of GROUP BY in SQL.

import pandas as pd

data = {
    'department': ['Eng', 'Eng', 'HR', 'HR', 'Eng'],
    'name':       ['Alice', 'Bob', 'Carol', 'Dave', 'Eve'],
    'salary':     [90000, 95000, 70000, 72000, 88000],
}
df = pd.DataFrame(data)

print(df.groupby('department')['salary'].mean())

Output:

department
Eng    91000.0
HR     71000.0
Name: salary, dtype: float64

Apply multiple aggregation functions at once using agg():

print(df.groupby('department')['salary'].agg(['mean', 'min', 'max', 'count']))

Output:

               mean    min    max  count
department
Eng        91000.0  88000  95000      3
HR         71000.0  70000  72000      2

Merging and Concatenating DataFrames

Merging (like SQL JOIN)

Use pd.merge() to combine two DataFrames on a shared key column:

import pandas as pd

employees = pd.DataFrame({
    'emp_id':  [1, 2, 3],
    'name':    ['Alice', 'Bob', 'Carol'],
    'dept_id': [10, 20, 10],
})
departments = pd.DataFrame({
    'dept_id':   [10, 20],
    'dept_name': ['Engineering', 'HR'],
})

merged = pd.merge(employees, departments, on='dept_id')
print(merged)

Output:

   emp_id   name  dept_id    dept_name
0       1  Alice       10  Engineering
1       2    Bob       20           HR
2       3  Carol       10  Engineering

The default is an inner join (only rows with a matching key in both DataFrames). Control the join type with how:

how='left' — all rows from the left DataFrame, matched rows from the right
how='right' — all rows from the right DataFrame, matched rows from the left
how='outer' — all rows from both DataFrames; NaN where there is no match

Concatenating (stacking DataFrames)

Use pd.concat() to stack DataFrames with the same columns vertically:

import pandas as pd

q1 = pd.DataFrame({'name': ['Alice', 'Bob'],   'sales': [120, 95]})
q2 = pd.DataFrame({'name': ['Carol', 'Dave'],  'sales': [110, 130]})

combined = pd.concat([q1, q2], ignore_index=True)
print(combined)

Output:

    name  sales
0  Alice    120
1    Bob     95
2  Carol    110
3   Dave    130

ignore_index=True resets the row index so it runs from 0 to 3 instead of repeating 0 and 1 from each source DataFrame.

Descriptive Statistics

pandas makes it easy to compute summary statistics on an entire DataFrame or individual columns:

import pandas as pd

data = {
    'name':  ['Alice', 'Bob', 'Carol', 'Dave'],
    'age':   [25, 30, 28, 35],
    'score': [88.5, 92.0, 79.5, 95.0],
}
df = pd.DataFrame(data)

print(df[['age', 'score']].describe())

Output:

             age      score
count   4.000000   4.000000
mean   29.500000  88.750000
std     4.203173   6.714412
min    25.000000  79.500000
25%    27.250000  86.250000
50%    29.000000  90.250000
75%    31.250000  92.750000
max    35.000000  95.000000

Individual statistics are also available as direct methods: df['score'].mean(), df['score'].std(), df['score'].median(), df['age'].max().

Basic Data Visualization

pandas integrates with Matplotlib so you can plot directly from a DataFrame with the .plot() method. Install Matplotlib first if you have not already: pip install matplotlib.

Line chart

import pandas as pd
import matplotlib.pyplot as plt

data = {
    'month':   ['Jan', 'Feb', 'Mar', 'Apr', 'May'],
    'revenue': [15000, 18000, 16500, 21000, 23000],
}
df = pd.DataFrame(data)

df.plot(kind='line', x='month', y='revenue', marker='o', title='Monthly Revenue')
plt.ylabel('Revenue ($)')
plt.tight_layout()
plt.show()

Bar chart

df.plot(kind='bar', x='month', y='revenue', color='steelblue', title='Monthly Revenue')
plt.ylabel('Revenue ($)')
plt.xticks(rotation=0)
plt.tight_layout()
plt.show()

Common values for the kind parameter:

`kind`	Chart type
`'line'`	Line chart
`'bar'`	Vertical bar chart
`'barh'`	Horizontal bar chart
`'hist'`	Histogram
`'box'`	Box plot
`'scatter'`	Scatter plot (requires `x` and `y`)
`'pie'`	Pie chart

For more advanced plotting, see the Matplotlib Introduction and Matplotlib Scatter Plot chapters.

Common Gotchas

SettingWithCopyWarning: When you filter a DataFrame and then try to modify the result, pandas may warn you that you are modifying a copy rather than the original. Use .copy() to be explicit:

# Safe: work on an explicit copy
young = df[df['age'] < 30].copy()
young['group'] = 'junior'

Chained indexing: df['col'][0] = value may or may not modify the original DataFrame. Always use df.loc[0, 'col'] = value for assignment.

Column names with spaces: If a column is named "first name", you must use bracket notation — df['first name'] — not dot notation (df.first name is a syntax error).

Integer vs. float after fillna: Filling NaN in an integer column with a number promotes the column to float64 because NaN is a float. Use pd.Int64Dtype() (nullable integer) if you need to keep integer semantics with missing values.

NumPy Tutorial — the array library that pandas is built on top of
Matplotlib Introduction — create publication-quality plots from your data
Matplotlib Scatter Plot — visualise relationships between two numeric variables
Data Distribution — understand how values are spread across a dataset
Scale — normalise and standardise numeric features before modelling
Categorical Data — encode non-numeric columns for machine learning
Train/Test Split — split a pandas DataFrame into training and test sets
SciPy Tutorial — statistical and scientific functions that complement pandas

Installing pandas

The Two Core Data Structures

Series

DataFrame

Reading Data from Files

Reading a CSV file

Reading an Excel file

Writing data back to a file

Selecting Data

Selecting columns

Selecting rows with loc and iloc

Filtering Rows

Adding and Modifying Columns

Sorting Data

Handling Missing Values

Filling missing values

Dropping rows with missing values

Grouping Data

Merging and Concatenating DataFrames

Merging (like SQL JOIN)

Concatenating (stacking DataFrames)

Descriptive Statistics

Basic Data Visualization

Line chart

Bar chart

Common Gotchas

Related Chapters

Selecting rows with `loc` and `iloc`