PYTHON / STRINGS
Unicode, encoding, and bytes
Convert between str and bytes deliberately with encode/decode, predict byte lengths, and diagnose UnicodeDecodeError and mojibake.
What you will learn
- Distinguish a str (code points) from bytes (integers 0-255) and know which you hold
- Use s.encode('utf-8') and b.decode('utf-8') and read UTF-8 byte lengths
- Explain why len(str) and len(bytes) differ for non-ASCII text
- Pick errors='replace', 'ignore' or 'backslashreplace' when strict decoding fails
Understanding Unicode, encoding, and bytes
A Python str is a sequence of Unicode code points, not a sequence of bytes. 'café' has four code points, and the last one is U+00E9, the integer 233 that Unicode assigns to LATIN SMALL LETTER E WITH ACUTE. Nothing about that str says how the character should be stored in a file or sent over a socket; the code point is an abstract number, and indexing, slicing and len() all work in units of those numbers.
A bytes object is the other thing: a sequence of integers from 0 to 255. Encoding is the mapping from code points to bytes, and decoding is the reverse. UTF-8 is variable width, so code points below 128 take one byte, U+0080 through U+07FF take two, most of the rest of the common characters take three, and emoji and other astral characters take four. That is why 'café'.encode('utf-8') is five bytes long while the str is four characters long, and why indexing into UTF-8 bytes does not line up with indexing into the str.
The critical asymmetry is that bytes carry no record of which encoding produced them. When you decode you are supplying a guess, and a wrong guess fails in one of two ways. If the bytes are not a legal sequence for the codec you named, you get a UnicodeDecodeError; if they happen to be legal but mean something else, you get mojibake, silently wrong text such as 'naïve'. latin-1 never raises, because all 256 byte values map to code points, which makes it a great way to hide a bug and a bad default.
text = "café"
print(len(text))
utf8 = text.encode("utf-8")
print(utf8)
print(len(utf8))
latin1 = text.encode("latin-1")
print(latin1, len(latin1))
print(utf8.decode("utf-8"))
try:
utf8.decode("ascii")
except UnicodeDecodeError as e:
print("failed:", e.reason)
print(ord("é"), hex(ord("é")))str is code points and bytes is raw octets; encode and decode are the only bridge, and decoding always requires knowing the encoding the bytes were written with.
Worked examples
Mojibake and how to undo it
Shows what happens when UTF-8 bytes are decoded as latin-1, and that the damage is reversible.
original = "naïve"
wire = original.encode("utf-8")
print(wire)
broken = wire.decode("latin-1")
print(broken)
repaired = broken.encode("latin-1").decode("utf-8")
print(repaired)Example explained
Line 1U+00EF (ï) becomes the two bytes C3 AF under UTF-8, so the bytes object is one longer than the str.
Line 2Decoding as latin-1 maps each byte to its own code point, giving à (0xC3) and ¯ (0xAF) instead of ï.
Line 3No exception is raised, because every byte value is valid latin-1; the failure is silent corruption.
Line 4Re-encoding the mojibake with latin-1 recovers the original bytes exactly, so decoding as UTF-8 restores the text.
len() counts code points, not characters or bytes
Contrasts a precomposed accent, a decomposed accent, and an emoji to show what len() actually measures.
import unicodedata
composed = "\u00e9"
decomposed = "e\u0301"
print(composed == decomposed, len(composed), len(decomposed))
print(unicodedata.normalize("NFC", decomposed) == composed)
emoji = "\U0001f600"
print(len(emoji), len(emoji.encode("utf-8")))Example explained
Line 1Both strings display as é, but one is a single code point and the other is 'e' plus a combining acute accent.
Line 2== compares code points, so visually identical text can be unequal; normalize('NFC', ...) collapses them to the composed form first.
Line 3The emoji is one code point above U+FFFF, so len() is 1 while UTF-8 needs four bytes for it.
Slicing bytes and the errors argument
Demonstrates that byte slices can cut a character in half, and how error handlers respond.
data = "Zürich".encode("utf-8")
print(data[0], data[1])
print(data[1:2])
print(data.decode("utf-8", errors="replace"))
print(data[:2].decode("utf-8", errors="replace"))
print("Zürich".encode("ascii", errors="ignore"))
print("Zürich".encode("ascii", errors="backslashreplace"))Example explained
Line 1Indexing bytes yields ints (90 is 'Z', 195 is the first byte of ü), while slicing yields a bytes object.
Line 2data[:2] cuts ü in half, so errors='replace' emits U+FFFD, the replacement character, for the truncated sequence.
Line 3errors='ignore' on encode drops ü entirely, which silently loses data and is rarely what you want.
Line 4errors='backslashreplace' writes the literal ASCII text \xfc, which is why the repr shows a doubled backslash.
Important notes
open() uses locale.getpreferredencoding() when you omit encoding=, so the same script can read a file correctly on one machine and crash on another; pass encoding='utf-8' explicitly.
Comparing str and bytes never raises but never matches either: 'a' == b'a' is False, so a stray bytes value can make a condition quietly always false.
Common mistakes
Calling str(some_bytes) instead of some_bytes.decode('utf-8'): you get the string "b'caf\\xc3\\xa9'" including the b and the escapes, and it silently poisons everything downstream.
Using errors='ignore' to make a UnicodeDecodeError go away: the traceback disappears but characters are permanently deleted from the data.
Assuming len() of a str equals the number of bytes needed, then slicing bytes at that offset: multi-byte characters get cut in half and decoding fails mid-stream.
Try it yourself
Change, predict, then run
Take the string 'Grüße, 世界', print len() of the str and of its UTF-8 encoding, then decode those same bytes as latin-1 and print the result to see exactly which characters turn into mojibake.
Open the Python workspaceCheck your understanding
A file holds the single character 'é' stored with latin-1 (one byte, 0xE9). Your code opens it without specifying an encoding on a UTF-8 system and calls read(). What happens?
- UnicodeDecodeError, because 0xE9 announces a multi-byte sequence but no continuation bytes follow
- It reads 'é' correctly, because Python inspects the bytes and detects the encoding
- It reads two replacement characters, since UTF-8 always needs two bytes for an accented letter
- It reads 'é', the classic mojibake for accented characters
Show answer
In UTF-8, a byte in the 0xE0-0xEF range is the start of a three-byte sequence, so a lone 0xE9 at end of file is invalid and strict decoding raises. Option 3 describes the opposite error, UTF-8 bytes read as latin-1, which cannot happen here because there is only one byte and latin-1 is the encoding that wrote it; note also that latin-1 decoding never raises, since all 256 byte values are valid.