How to get line count of a large file cheaply in Python?

You can count the lines of a large file efficiently in Python by reading it line by line without loading the entire file into memory.

Count the lines of a file in Python

def line_count(file_path):
    with open(file_path, "r") as f:
        return sum(1 for _ in f)

This function reads the file line by line and returns the number of lines in the file. Since it reads the file one line at a time, it can handle large files efficiently without loading the entire file into memory. Note that lines are counted based on newline characters, so a file without a trailing newline will still count its final line.

You can also use the wc command via the subprocess module to get the line count of a large file cheaply in Python:

Count the lines of a file in Python using subprocess

import subprocess

def line_count(file_path):
    result = subprocess.run(['wc', '-l', file_path], capture_output=True, text=True)
    return int(result.stdout.split()[0])

This function uses the wc command, which is available on most UNIX-like systems and provides a fast way to count the number of lines in a file. Using subprocess.run is the modern, secure alternative to the deprecated os.popen, and passing arguments as a list safely handles file paths containing spaces.