Python open() with UTF-8: Read and Write Unicode Files
Open files as UTF-8 in Python with open(path, encoding="utf-8"): read and write Unicode text, fix UnicodeDecodeError on Windows, handle BOM and CSV.
Short answer: pass encoding="utf-8" to open() every time you read ("r"), write ("w"), or append ("a") a text file: open(path, encoding="utf-8"). If you leave it out, Python picks a platform-dependent default encoding, which on Windows is often cp1252 instead of UTF-8, so the same script can work on Linux or macOS and crash with a UnicodeDecodeError on Windows.
To read a file in Unicode (UTF-8) encoding in Python, you can use the built-in open() function, specifying the encoding as "utf-8". Here's an example:
Read a file in Unicode (UTF-8) encoding in Python using open() function
with open("file.txt", "r", encoding="utf-8") as f:
text = f.read()
print(text)Watch a video course: Python - The Practical Guide
To write a file in Unicode (UTF-8) encoding in Python, you can use the built-in open() function with the 'w' mode and specifying the encoding as "utf-8". Here's an example:
Write a file in Unicode (UTF-8) encoding in Python using open() function
with open("file.txt", "w", encoding="utf-8") as f:
f.write("Hello, world!")Please note that in the above example the file will be created if it does not exist, and if the file already exists, its content is truncated (erased) before writing.
Writing and reading non-ASCII text
"Hello, world!" is plain ASCII, so it looks the same in almost any encoding. The encoding argument really matters once the text contains accented letters, other alphabets, or emoji. This example writes such text and reads it back:
# Write Unicode text, then read it back
with open("notes.txt", "w", encoding="utf-8") as f:
f.write("café, naïve, 日本語, emoji: 🐍\n")
with open("notes.txt", "r", encoding="utf-8") as f:
print(f.read(), end="")Output:
café, naïve, 日本語, emoji: 🐍To add text to the end of an existing file without erasing it, use the "a" (append) mode, again with encoding="utf-8". Iterating over the file object reads it line by line, which is memory-friendly for large files:
with open("notes.txt", "a", encoding="utf-8") as f:
f.write("Ünïcödé line\n")
with open("notes.txt", encoding="utf-8") as f:
for line in f:
print(line.rstrip("\n"))Output:
café, naïve, 日本語, emoji: 🐍
Ünïcödé lineThe with statement closes the file automatically, even if an error occurs. For more on file modes, see Python file handling.
Why you should always pass encoding="utf-8"
When encoding is omitted, open() uses the locale's preferred encoding (locale.getpreferredencoding(False)). On most Linux and macOS systems that is UTF-8, but on Windows it is usually a legacy code page such as cp1252. Reading a UTF-8 file with the wrong codec either raises an error or silently produces garbled text ("mojibake"). The following example forces cp1252 to show what happens on a typical Windows machine:
# Simulate the Windows default (cp1252) reading a UTF-8 file
with open("quotes.txt", "w", encoding="utf-8") as f:
f.write("Smart quotes: “hello”\n")
try:
with open("quotes.txt", encoding="cp1252") as f:
f.read()
except UnicodeDecodeError as e:
print(type(e).__name__ + ":", e)Output:
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 24: character maps to <undefined>Ways to avoid this:
- Pass
encoding="utf-8"explicitly. This is the most portable fix and works on every Python 3 version. - Enable Python's UTF-8 Mode for the whole interpreter, so that
open()defaults to UTF-8: runpython -X utf8 script.pyor set the environment variablePYTHONUTF8=1. PEP 686 makes UTF-8 Mode the default starting with Python 3.15. - Find missing
encodingarguments withpython -X warn_default_encoding script.py(Python 3.10+), which emits anEncodingWarningfor everyopen()call that relies on the default.
Files with a BOM: use "utf-8-sig"
Some Windows tools (for example Notepad in older versions, or Excel when exporting "CSV UTF-8") put a byte order mark (BOM) at the start of the file. With encoding="utf-8" the BOM shows up as an invisible '' character at the start of the text, which breaks comparisons like header == "name". The utf-8-sig codec strips it when reading (and works fine on files without a BOM):
# A file saved with a byte order mark (BOM)
with open("bom.txt", "w", encoding="utf-8-sig") as f:
f.write("name\n")
with open("bom.txt", encoding="utf-8") as f:
print(repr(f.read())) # BOM leaks in as ''
with open("bom.txt", encoding="utf-8-sig") as f:
print(repr(f.read())) # BOM strippedOutput:
'name\n'
'name\n'When writing, utf-8-sig adds a BOM. Use it only if the consumer needs one (for example, so Excel recognizes a CSV file as UTF-8); otherwise plain utf-8 is the standard choice.
Handling invalid bytes with errors=
If a file is not actually valid UTF-8, reading it raises UnicodeDecodeError by default (errors="strict"). The errors argument lets you choose another behavior: "replace" substitutes the replacement character � (U+FFFD), "ignore" drops the bad bytes, and "backslashreplace" keeps them as escape sequences. If you know the file's real encoding, reading it with that encoding is the better fix:
# Bytes that are not valid UTF-8 (0xE9 is "é" in Latin-1)
with open("latin1.txt", "wb") as f:
f.write(b"caf\xe9\n")
with open("latin1.txt", encoding="utf-8", errors="replace") as f:
print(repr(f.read())) # invalid byte -> U+FFFD
with open("latin1.txt", encoding="latin-1") as f:
print(repr(f.read())) # correct if you know the real encodingOutput:
'caf�\n'
'café\n'Reading and writing UTF-8 with pathlib
For short files, pathlib.Path offers one-line helpers. read_text() and write_text() accept the same encoding argument as open(), and have the same platform-dependent default if you omit it:
from pathlib import Path
path = Path("greeting.txt")
path.write_text("Привет, мир!\n", encoding="utf-8")
print(path.read_text(encoding="utf-8"), end="")Output:
Привет, мир!UTF-8 CSV files
The csv module works on text files, so the encoding is set in open(). Also pass newline="", as the csv documentation recommends, so that line endings inside fields are handled correctly and no blank rows appear on Windows:
import csv
rows = [["name", "city"], ["José", "São Paulo"], ["Zoë", "Zürich"]]
with open("people.csv", "w", encoding="utf-8", newline="") as f:
csv.writer(f).writerows(rows)
with open("people.csv", encoding="utf-8", newline="") as f:
for row in csv.reader(f):
print(row)Output:
['name', 'city']
['José', 'São Paulo']
['Zoë', 'Zürich']Reading bytes and decoding manually
Opening a file in binary mode ("rb") gives you raw bytes with no decoding at all (binary mode does not accept an encoding argument). You can then decode them yourself with bytes.decode("utf-8"). This is useful when you need to inspect the raw data or detect the encoding before decoding:
with open("notes.txt", "rb") as f:
data = f.read() # raw bytes
print(data[:15])
print(data.decode("utf-8"), end="")Output:
b'caf\xc3\xa9, na\xc3\xafve, '
café, naïve, 日本語, emoji: 🐍
Ünïcödé lineCommon pitfalls
- Forgetting
encodingon one of the calls. Writing withencoding="utf-8"and reading without it (or the other way round) works on your machine and fails on another one. - Using
"w"when you meant"a". Mode"w"erases the existing content first. - Python 2 code. In Python 2 the built-in
open()has noencodingparameter; legacy code usedio.open(path, encoding="utf-8")instead. In Python 3,io.openis the same function asopen, so just useopen(). - Decoding the same text twice.
open(..., encoding="utf-8")already returnsstr; calling.decode()on it raisesAttributeError. Onlybytes(from"rb"mode) need decoding.
To go further, read the W3docs chapters on reading files in Python and writing and creating files in Python.