Python UnicodeDecodeError Reading a File: cp1251 vs UTF-8

If Python prints Хлеб where the word "Хлеб" should be, or crashes with UnicodeDecodeError while reading a file, both symptoms have one cause: the file was written in one encoding and Python reads it in another. The fix is one argument to open() — encoding="cp1251" for Excel and legacy Russian exports, encoding="utf-8" for everything else.

Are mojibake and UnicodeDecodeError the same bug?

Yes. The root cause is identical — Python assumed the wrong codec — and the symptom depends only on whether the file's bytes happen to be legal in that codec. Single-byte encodings such as cp1251 and koi8-r assign a character to almost every one of the 256 possible byte values, so they almost never fail; they hand you garbage silently. UTF-8 is stricter: some byte sequences are illegal, and on the first illegal one Python raises.

A UTF-8 file read as cp1251 — no error, no warning:

with open("notes.txt", "w", encoding="utf-8") as f:
    f.write("Хлеб;45.90")

with open("notes.txt", encoding="cp1251") as f:
    print(f.read())

Хлеб;45.90

The opposite direction, a cp1251 file read as UTF-8, is broken on purpose — it raises:

with open("prices.txt", "w", encoding="cp1251") as f:
    f.write("Хлеб;45.90\nМолоко;89.50\nСыр;349.00\n")

with open("prices.txt", encoding="utf-8") as f:
    text = f.read()
print(text)

Traceback (most recent call last):
  File "read_prices.py", line 5, in <module>
    text = f.read()
           ^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd5 in position 0: invalid continuation byte

The crash is the lucky outcome: it stops the program at once, while mojibake travels quietly into your database, your report and your customer.

What encoding does open() use by default?

There is no fixed default. In text mode open() uses locale.getpreferredencoding(False), which depends on the operating system and its regional settings: cp1251 on a Russian Windows install, normally UTF-8 on Linux and macOS. This is why "works on my machine" proves nothing here — it describes the author's locale, not the file. On Russian Windows the prompt shows it:

>>> import locale
>>> locale.getpreferredencoding(False)
'cp1251'

Two ways to remove the guess:

Approach	What it does	When to use it
encoding="utf-8" in every open()	Removes the guess at one call site	Always — this is the working habit
PYTHONUTF8=1 or python -X utf8 script.py	Turns on UTF-8 mode for the whole process	Someone else's script you would rather not edit

The first is stronger: it travels with the code, an environment variable stays on one machine. Basic forms are in the handbook under reading and writing text files.

Reading the message: codec, byte, position, reason

The last line of that traceback carries four facts, all four useful:

• 'utf-8' codec — the codec Python tried. Your assumption, not a property of the file.
• byte 0xd5 — the byte that broke. 0xD5 is the letter "Х" in cp1251.
• in position 0 — the offset inside the decoded chunk. Position 0 means the file is "wrong" from its first character, not in one random line in the middle.
• invalid continuation byte — in UTF-8 this byte must start a multi-byte sequence and be followed by a 10xxxxxx byte, and it was not.

The same facts live on the exception object, which matters when you process many files and log failures instead of squinting at them:

with open("prices.txt", "w", encoding="cp1251") as f:
    f.write("Хлеб;45.90")

try:
    with open("prices.txt", encoding="utf-8") as f:
        text = f.read()
except UnicodeDecodeError as e:
    print("codec:   ", e.encoding)
    print("position:", e.start)
    print("byte:    ", hex(e.object[e.start]))
    print("reason:  ", e.reason)

codec:    utf-8
position: 0
byte:     0xd5
reason:   invalid continuation byte

Catch UnicodeDecodeError specifically, not a bare except, or a filename typo lands in the same branch — see catch a specific exception.

cp1251, koi8-r and UTF-8: telling them apart by symptom

Mojibake is not random. Every "real → assumed" pair has its own handwriting, so you can often name the encoding without running anything. Here is the word "Хлеб":

File written in	Read as	What you see
UTF-8	cp1251	Хлеб
UTF-8	koi8-r	п╔п╩п╣п╠
koi8-r	cp1251	иМЕВ
cp1251	UTF-8	UnicodeDecodeError
cp1251	cp1251	Хлеб

Capital Р and С alternating with one stray symbol each is almost always UTF-8 read as cp1251: a Cyrillic character in UTF-8 starts with 0xD0 or 0xD1, and those bytes are Р and С in cp1251. Box-drawing noise like п╔п╩ is UTF-8 read as koi8-r; wrong-but-real Russian letters are koi8-r read as cp1251.

Is errors='replace' a fix or a diagnostic?

A diagnostic. errors="replace" tells open() not to raise and to substitute � for every run of bytes it could not decode — here exactly one per Cyrillic letter — so you see the shape of the damage instead of only its first byte:

with open("prices.txt", "w", encoding="cp1251") as f:
    f.write("Хлеб;45.90\nМолоко;89.50\nСыр;349.00")

with open("prices.txt", encoding="utf-8", errors="replace") as f:
    print(f.read())

����;45.90
������;89.50
���;349.00

That picture answers the main question at a glance: only the letters broke, digits and semicolons survived. The file is not corrupt — it is simply not UTF-8, and its structure is the one you expected. Had the digits gone too, you would be looking at an xlsx or an archive, not text. Do not ship errors="replace": replaced characters cannot be recovered. Codecs and manual bytes.decode() are covered in text encoding and decoding.

How does encoding='utf-8-sig' differ from 'utf-8'?

utf-8-sig is the same UTF-8 with one extra ability: it strips the BOM, three service bytes EF BB BF at the start of the file. Excel writes them when you save as "CSV UTF-8", as do some Windows editors. Plain utf-8 reads such a file without error, but the first character becomes an invisible \ufeff and exact string matching breaks.

with open("prices.txt", "w", encoding="utf-8-sig") as f:
    f.write("45.90\n89.50")

with open("prices.txt", encoding="utf-8") as f:
    first = f.readline().strip()

print(repr(first))
try:
    print(float(first))
except ValueError as e:
    print("ValueError:", e)

'\ufeff45.90'
ValueError: could not convert string to float: '\ufeff45.90'

That is how a BOM becomes a failure on the first line and on no other; the other causes of the same exception are in could not convert string to float. The fix is encoding="utf-8-sig": it removes a BOM if there is one and changes nothing if there is not, a sensible default for other people's CSV files.

How do you write a CSV that Excel opens without mojibake?

The mirror image of the same problem. Excel on Windows opens a CSV in the system encoding and shows honest UTF-8 as mojibake until it sees a BOM, so a file bound for Excel needs encoding="utf-8-sig" on write. Plain encoding="utf-8" puts the same Хлеб into the spreadsheet.

import csv

rows = [["товар", "цена"], ["Хлеб", "45.90"], ["Молоко", "89.50"]]

with open("report.csv", "w", encoding="utf-8-sig", newline="") as f:
    csv.writer(f, delimiter=";").writerows(rows)

print(open("report.csv", "rb").read(3))

b'\xef\xbb\xbf'

Those three bytes are the BOM. newline="" is mandatory for the csv module: without it Windows doubles the line endings.

How do you detect the encoding of an unknown file?

There is no reliable way to detect an encoding from content alone: the same bytes are valid text in several single-byte encodings at once, and a text file stores no note about itself. Try a short candidate list and look at the results yourself. For Russian files four cover almost everything: utf-8-sig, cp1251, koi8-r, cp866.

from pathlib import Path

Path("prices.txt").write_bytes("Хлеб;45.90".encode("cp1251"))
raw = Path("prices.txt").read_bytes()

for codec in ("utf-8-sig", "cp1251", "koi8-r"):
    try:
        text = raw.decode(codec)
    except UnicodeDecodeError:
        print(f"{codec:10} fails")
    else:
        print(f"{codec:10} {text}")

utf-8-sig  fails
cp1251     Хлеб;45.90
koi8-r     уКЕА;45.90

Look at koi8-r: it did not fail, it produced уКЕА. Single-byte codecs almost never fail — koi8-r and cp866 define all 256 byte values, cp1251 leaves exactly one undefined (0x98) — so "loop until it stops crashing" rules out little more than UTF-8; a human picks the right line. Automating it means charset-normalizer or chardet, third-party libraries returning a probability, not a guarantee.

The one-line rule: always pass encoding explicitly

Pass encoding in every open() that reads text, even when it already works. One line removes a class of bugs that only reproduce on someone else's machine: utf-8 for your own files, utf-8-sig for other people's CSV, cp1251 for legacy Russian exports.

The writing side has a symmetric knob. If you save a dict with json.dump() and find \u0425\u043b\u0435\u0431 in the file instead of Cyrillic letters, encoding is not to blame — that is ensure_ascii, explained in Cyrillic saved as Unicode escapes.

Common mistakes

1. Calling open() with no encoding. The most common cause of the whole problem. Fix: open(path, encoding="utf-8") everywhere you read text.
2. Deciding the file is corrupt and asking for a new copy. You will get an identical one. Fix: read the first bytes in "rb" mode and check for \xd5 versus \xd0\xa5.
3. Shipping errors="replace". The program stops crashing and loses data silently instead. Fix: investigate with it, then set the correct encoding.
4. Re-saving the file in an editor instead of fixing the code. That works once; the next export arrives in cp1251 again.
5. Reading Excel's "CSV UTF-8" as utf-8. No error, but the first header stops matching anything. Fix: encoding="utf-8-sig".
6. Confusing reading with writing. UnicodeDecodeError happens on read; UnicodeEncodeError ('charmap' codec can't encode character) on write and when printing to a Windows console.
7. Wrapping the read in except Exception. FileNotFoundError, PermissionError and a path typo land in the same branch. Fix: catch UnicodeDecodeError on its own.

Practice it in the browser

Encoding itself is not something you drill, but everything right after f.read() is ordinary string work: split, clean, build a dict — the step where invisible characters like a BOM surface. Invisible characters break source files too, not just data: tabs mixed with spaces in the indentation raise a TabError, caught with the same repr() trick.

• Parse a settings string — turn key=value lines into a dict, the shape of reading a config file.
• Tidy up the spaces — clean the stray whitespace exports ship with.
• More free string tasks: the filtered catalogue.

File names from that export arrive with an extension, and trimming it has its own trap: rstrip(".txt") eats letters, covered in removing a file extension.

In short

• Mojibake and UnicodeDecodeError are one bug: Python assumed the wrong encoding.
• Single-byte codecs (cp1251, koi8-r) almost never fail and corrupt text silently; UTF-8 fails loudly, which is more useful.
• open() has no fixed default — it takes one from the locale, so Windows and Linux differ on identical code.
• All four parts of the message matter: codec, byte, position, reason.
• errors="replace" shows the damage; it does not repair it.
• utf-8-sig strips Excel's BOM on read and writes it back for Excel.
• Content-based detection is never certain — try candidates and read the output yourself.