Character Encoding History
From Baudot to ASCII to Unicode
Character encoding went from total chaos in the 1950s - every manufacturer inventing proprietary mappings - to ASCII in 1963, a patchwork of Extended ASCII standards in the 1980s, and finally Unicode in 1991. Each step solved real problems and created new ones. The throughline is a 60-year push toward one universal mapping that handles every human writing system.
The Pre-Standardization Chaos
Before 1963, there was no agreement on how to represent text in binary. IBM used BCD (Binary Coded Decimal) on its 704 and 7090 mainframes. CDC used a 6-bit display code. Teletype machines used 5-bit Baudot code. A text file created on one machine was gibberish on another.
The root cause was simple: no coordination. Each manufacturer optimized for its own hardware constraints, and there was no economic incentive to interoperate. The US military and telecommunications industry pushed for standardization because they needed to exchange messages across equipment from different vendors.
ASCII: The 7-Bit Standard (1963)
The American Standards Association (now ANSI) published X3.4-1963, assigning 128 code points (0-127) to a fixed set of characters. The mapping was intentional: uppercase letters at 65-90, digits at 48-57, control characters at 0-31 and 127. The bit-level layout made sorting, case conversion, and digit extraction trivial for hardware of that era.
ASCII solved the interoperability problem for English text. Unix, ARPANET (the precursor to the internet), email (RFC 822), and HTTP all built on ASCII as their text foundation. But 128 characters cannot represent French accents, German umlauts, or any non-Latin script. The timeline covers the key milestones.
Extended ASCII: The 8-Bit Fragmentation (1970s-1990s)
Extended ASCII used an 8th bit to double the code space to 256 characters. The first 128 positions stayed the same as ASCII; the upper 128 varied by region and vendor. This is where fragmentation began.
ISO 8859-1 (Latin-1) covered Western European languages. ISO 8859-5 covered Cyrillic. CP437 provided box-drawing characters for DOS. Windows-1252 extended Latin-1 with smart quotes and other typographic characters. Each was internally consistent, but position 0x80 might be C-cedilla in one encoding and a box-drawing corner in another.
The result: mojibake. Open a Cyrillic text file with a Western European decoder, and every character above 127 renders as the wrong glyph. There was no reliable way to detect which encoding a file used, so applications guessed - and guessed wrong often enough to be a persistent source of bugs. You can compare encoding differences to see exactly how the same byte maps to different characters.
Unicode: One Registry to Rule Them All (1991-Present)
The Unicode Consortium published Unicode 1.0 in 1991 with the goal of assigning a unique code point to every character in every writing system. Version 16.0 (2024) defines over 154,000 characters across 168 scripts, including historical scripts like Sumerian cuneiform and modern additions like emojis.
Unicode itself is not an encoding - it is a character registry. The actual byte representation depends on the encoding form: UTF-8 (variable-width, 1-4 bytes), UTF-16 (2 or 4 bytes), or UTF-32 (fixed 4 bytes). UTF-8 won the web because it is backward-compatible with ASCII: any valid ASCII file is already valid UTF-8 with identical byte values for the first 128 code points.
Why UTF-8 Won
Three properties made UTF-8 the dominant encoding for the web (over 98% of web pages as of 2024). First, backward compatibility with ASCII means existing English-language content required zero changes. Second, variable-width encoding means common Latin characters use 1 byte while rarer scripts use more, keeping average file sizes smaller than fixed-width alternatives.
Third, UTF-8 is self-synchronizing. If you start reading bytes in the middle of a stream, you can always find the start of the next valid character without backtracking to the beginning. This property is critical for search engines, databases, and text processing tools that need to jump into arbitrary positions in large files.
The full history from telegraph codes to modern Unicode spans over 150 years. For a visual walkthrough, see the encoding timeline. For details on specific encoding standards, browse the character sets reference.