How UTF-8 Actually Works

1. The Byte Structure

UTF-8 uses leading bits in each byte to signal how many bytes form a single character. The pattern is rigid and unambiguous:

Code Point Range	Bytes	Byte Pattern	Available Bits
U+0000 to U+007F	1	0xxxxxxx	7
U+0080 to U+07FF	2	110xxxxx 10xxxxxx	11
U+0800 to U+FFFF	3	1110xxxx 10xxxxxx 10xxxxxx	16
U+10000 to U+10FFFF	4	11110xxx 10xxxxxx 10xxxxxx 10xxxxxx	21

Every leading byte starts with a specific bit prefix: 0 for single-byte (pure ASCII), 110 for two-byte, 1110 for three-byte, 11110 for four-byte. Continuation bytes always start with 10. No valid byte sequence is ambiguous. A decoder reading any byte in a stream can immediately tell whether it is a leading byte or a continuation byte by checking the first two bits.

2. Why ASCII Compatibility Matters

The first 128 code points (U+0000 through U+007F) encode as single bytes with a leading zero bit. U+0041 (A) becomes 0x41. U+000A (line feed) becomes 0x0A. Every valid ASCII file is already valid UTF-8 with zero modifications.

This backward compatibility is not accidental. Thompson and Pike designed it specifically so that existing C string functions (strlen, strchr, strcmp) would continue to work on ASCII data. No ASCII byte (0x00-0x7F) ever appears as part of a multi-byte sequence, because continuation bytes use the range 0x80-0xBF and leading bytes use 0xC0-0xFF. A strchr searching for / (0x2F) in a UTF-8 string will never false-match against a continuation byte of a Chinese character.

Compare this against the full ASCII table to verify: every code from 0 through 127 uses the same byte value in both encodings. Use the hex to ASCII converter to see the correspondence directly.

3. Encoding a Character Step by Step

Take the Euro sign, U+20AC. Its code point value is 8364 in decimal, which falls in the three-byte range (U+0800 to U+FFFF). Binary representation: 0010 0000 1010 1100.

Slot these 16 bits into the three-byte template 1110xxxx 10xxxxxx 10xxxxxx:

Code point: 0010 0000 1010 1100
Split:      0010 | 000010 | 101100
Template:   1110xxxx 10xxxxxx 10xxxxxx
Result:     11100010 10000010 10101100
Hex:        E2       82       AC

The Euro sign is stored as three bytes: E2 82 AC. Use the base converter to verify the binary-to-hex conversion. Every multi-byte character follows this same mechanical process: convert the code point to binary, split into groups matching the template, prefix each group with the appropriate leading bits.

4. Self-Synchronization

Drop into the middle of a UTF-8 byte stream at any arbitrary position. You can find the start of the next valid character without backtracking to the beginning of the file. If the current byte starts with 10, it is a continuation byte - skip forward until you find a byte that does not start with 10. That next byte is either a single-byte character (starts with 0) or the leading byte of a multi-byte sequence (starts with 110, 1110, or 11110).

This property makes UTF-8 safe for substring search, random file access, and parallel text processing. A search engine indexing a 10 GB text corpus can split it into chunks at arbitrary byte boundaries and process each chunk on a different machine. Each worker skips any partial character at the chunk boundary and synchronizes within at most 3 bytes.

Neither UTF-16 nor UTF-32 has this property in the same way. UTF-16 uses 2-byte code units, and a surrogate pair looks like two independent code units without surrounding context. UTF-32 is self-synchronizing but wastes 4 bytes per character regardless of value.

5. Overlong Encodings and Security

An overlong encoding represents a code point using more bytes than necessary. The NUL character (U+0000) should encode as the single byte 0x00. But the two-byte sequence C0 80 also decodes to U+0000 if a decoder blindly follows the template math. RFC 3629 explicitly forbids overlong encodings, and any conforming decoder must reject them.

This matters for security. Early web servers used C string functions that treat 0x00 as a string terminator. An attacker could send C0 80 to represent a NUL byte that passes through C string checks (because neither 0xC0 nor 0x80 is zero) but gets decoded as NUL by the application layer. Directory traversal attacks used overlong encodings of / and .. to bypass path validation filters. Microsoft IIS was famously vulnerable to this in 2001 (CVE-2001-0333).

Modern libraries (Python 3, Go, Rust, Java's StandardCharsets.UTF_8) reject overlong sequences by default. If you are writing a custom UTF-8 parser, always validate that each code point uses the minimum number of bytes.

6. BOM: Byte Order Mark

UTF-8 has no byte-order ambiguity (bytes are always processed left to right), so a BOM is unnecessary. Despite this, Windows tools (Notepad, PowerShell, Excel) often prepend three bytes EF BB BF (the UTF-8 encoding of U+FEFF) to files. This BOM breaks shell scripts on Unix (#!/bin/bash becomes \xEF\xBB\xBF#!/bin/bash, and the kernel does not recognize the shebang). It also breaks JSON parsers that expect the first byte to be { or [.

Strip the BOM with: sed -i '1s/^\xEF\xBB\xBF//' file.txt on Linux, or open the file in VS Code and change the encoding selector in the status bar to "UTF-8" (without BOM). Python handles BOM via the utf-8-sig codec: open('file.txt', encoding='utf-8-sig') strips a leading BOM automatically.

7. UTF-8 vs UTF-16 vs UTF-32

Property	UTF-8	UTF-16	UTF-32
Bytes per ASCII char	1	2	4
Bytes per CJK char	3	2	4
Bytes per emoji	4	4 (surrogate pair)	4
Endian-dependent	No	Yes (LE/BE)	Yes (LE/BE)
ASCII-compatible	Yes	No	No
Self-synchronizing	Yes	Partial	Yes
Null bytes in ASCII text	No	Yes (every other byte)	Yes (3 of 4 bytes)

UTF-16 uses less space than UTF-8 for CJK-heavy text (2 bytes vs 3 per character). Java and JavaScript use UTF-16 internally for strings because they predate UTF-8's dominance. Windows APIs also use UTF-16 (the W suffix functions like CreateFileW).

UTF-32 uses exactly 4 bytes per character regardless of value. Simple to index (character N is at byte offset 4N), but the space cost is prohibitive for storage and transmission. No major file format or protocol uses UTF-32 for interchange.

For files, network protocols, and databases, UTF-8 wins on three counts: smallest average size for mixed-script text, no endianness issues, and full ASCII compatibility. See the encoding history article for how this transition played out. Check the character sets reference for details on older encoding standards that UTF-8 replaced.

8. Detecting and Debugging Encoding Problems

When text displays as garbled characters, the underlying bytes are usually fine. The problem is that the reader assumes a different encoding than the writer used. Common symptoms:

• é instead of e (accented e) - UTF-8 bytes interpreted as ISO 8859-1
• e (single replacement character) instead of e - ISO 8859-1 bytes fed to a strict UTF-8 decoder
• “ instead of a left double quote - Windows-1252 smart quotes misread as UTF-8, then re-encoded

Diagnose by examining raw bytes with xxd file.txt | head or od -A x -t x1z file.txt. If the byte sequence for a character matches the UTF-8 pattern above, the file is UTF-8. If accented characters are single bytes in the range 0x80-0xFF, the file is likely ISO 8859-1 or Windows-1252. Use the text converter to see how the same text looks in different representations. Check the encoding comparison tool to see exactly where code pages diverge.

For a detailed troubleshooting guide with specific fix recipes for databases, email, and terminals, see the mojibake article.