History of Character Encoding

Samuel Morse and Alfred Vail demonstrated electrical telegraphy using a binary-like system of dots and dashes. Unlike later digital codes, Morse used variable-length sequences - E (the most frequent English letter) got a single dot, while less common letters got longer patterns. This frequency-based design inspired later compression algorithms but was too irregular for direct computer use.

Emile Baudot invented a 5-bit binary code for telegraph transmission, encoding 32 values per mode. To handle more than 32 characters, Baudot used a SHIFT mechanism toggling between "letters" and "figures" modes - a concept that survives in keyboard design today. The 5-bit limit (only 32 values per mode) drove the later decision to use 7 bits for ASCII, which eliminated the need for mode shifting.

Bob Bemer, an IBM programmer who had cataloged over 60 incompatible character codes used by different manufacturers, submitted a proposal to the American Standards Association (ASA) for a unified encoding. His design drew on the 6-bit binary codes used in the IBM 704 but expanded to 7 bits to accommodate both uppercase and lowercase letters alongside digits and control codes.

The ASA published ASA X3.4-1963, the first ASCII standard. This version defined 128 code points but assigned only 97 characters - notably missing lowercase letters (positions 96-122 were left unassigned). The up-arrow and left-arrow occupied positions that would later become caret (^) and underscore (_) in the 1967 revision.

Kenneth Knowlton at Bell Labs created images using the character set of an IBM 7094 mainframe. His collaboration with artist Leon Harmon produced "Studies in Perception" (1966) - photographs reproduced as grids of printed characters. This technique evolved into ASCII art, which remains active today with tools like image-to-ASCII converters and text art generators.

A major revision of ASCII was completed and accepted as a standard but never published or deployed. ASCII-1965 closely resembled the modern character set, adding lowercase letters, but procedural delays prevented its release. The changes were rolled into the 1967 revision instead.

The ASA released USAS X3.4-1967, the revision that established the ASCII table as it exists today. This version added lowercase letters, replaced the up-arrow with caret (^), replaced the left-arrow with underscore (_), and finalized all 128 character assignments that remain unchanged over 55 years later.

A minor revision formalized the standard as ANSI X3.4-1968 without changing any graphic characters. This is the version number still referenced in most technical specifications, RFCs, and programming language documentation when citing "the ASCII standard."

President Lyndon B. Johnson signed a memorandum on March 11, 1968, requiring all computers purchased by the US federal government to support ASCII. This forced IBM to add ASCII support alongside EBCDIC on its mainframes for government contracts, accelerating ASCII adoption across the entire industry.

A revision clarified ambiguities in the 1967 and 1968 editions, particularly around control character semantics. Several control codes like DC1 (XON) and DC3 (XOFF) received formal definitions for software flow control - the protocol that lets a slow printer tell a fast computer to pause output.

IBM shipped the original PC with Code Page 437, a 256-character encoding that extended ASCII with box-drawing characters, mathematical symbols, and accented letters. CP437 became the first widely deployed extended ASCII encoding and defined the character repertoire that DOS applications used for the next 15 years.

Microsoft released Windows 1.0, which introduced its own character encoding that Microsoft confusingly labeled "ANSI." This encoding (later standardized as Windows-1252) was not based on any actual ANSI standard. The misleading "ANSI" label persists in the Windows API to this day - CreateWindowA uses the "ANSI" code page, while CreateWindowW uses UTF-16.

The final revision of the ASCII standard added formal definitions for the C1 control codes (0x80-0x9F), which were used by ISO 6429 terminal protocols. The standard document number ANSI X3.4-1986 is technically the current edition - no further revisions have been issued because Unicode has subsumed ASCII's role.

MS-DOS 3.3 added code page switching, allowing users to load different character sets for the 128-255 range. New code pages included CP850 (Western European), CP852 (Central European), CP855 (Cyrillic), and CP857 (Turkish). This was the first time DOS users could display non-English characters without third-party software, though switching code pages mid-session often caused display corruption.

Joe Becker at Xerox coined the term "Unicode" to describe a proposed "unique, universal, and uniform" character encoding. Becker's initial draft proposed a 16-bit fixed-width encoding (65,536 characters), which he believed would be sufficient for all modern scripts. This 16-bit assumption later proved wrong, leading to the supplementary planes and UTF-8's variable-width design.

The Unicode Consortium released version 1.0 with 7,161 characters covering 24 scripts. The standard deliberately assigned ASCII's 128 characters to the same code points (U+0000 through U+007F), ensuring backward compatibility. Ken Thompson and Rob Pike invented UTF-8 the following year (1992) as a variable-width encoding that made this compatibility practical.

The IETF published RFC 1521 (MIME), which standardized how email systems declare character encodings using the Content-Type: text/plain; charset=us-ascii header. MIME also defined Base64 encoding for transmitting binary data through ASCII-only email channels - a workaround still used today for email attachments.

Unicode 2.0 expanded to 38,885 characters and introduced the surrogate pair mechanism for UTF-16, which allowed encoding beyond the original 16-bit limit (the Basic Multilingual Plane). This version added scripts including Tibetan, Canadian Syllabics, and several Indic scripts, making Unicode viable for most of Asia.

Released July 14, Unicode 5.0 reached 99,089 characters and added several historic scripts including Phags-pa (Mongolian empire), Phoenician (ancestor of Greek and Latin alphabets), and N'Ko (West African). By this point Unicode had become the dominant encoding for new software projects, though legacy systems still relied heavily on 8-bit encodings.

In December 2007, UTF-8 surpassed ASCII as the most common encoding on the World Wide Web, according to Google's web crawl data. The transition was gradual and mostly invisible because UTF-8 is byte-for-byte identical to ASCII for English text - existing ASCII pages became valid UTF-8 automatically when servers updated their Content-Type headers.

Unicode 8.0 added 7,716 characters including the first set of skin-tone emoji modifiers (Fitzpatrick scale), Georgian lari currency sign, and several historic scripts. Emoji had become a driving force for Unicode updates - the Consortium processed more emoji proposals than script additions for the first time.

Unicode 10.0 brought the total to 136,690 characters, adding the Bitcoin sign (U+20BF), 56 new emoji, and four new scripts (Zanabazar Square, Soyombo, Nushu, and Masaram Gondi). UTF-8 usage on the web had surpassed 90% by this point, making ASCII-only websites increasingly rare outside of specialized embedded systems.

Unicode 15.0 (September 2022) added 4,489 characters for a total of 149,186, including two new scripts (Kawi and Nagari) and 31 new emoji. ASCII's 128 original characters remain unchanged at their original code points (U+0000-U+007F) - the same positions assigned in 1991. The full ASCII table serves as the foundation layer of every Unicode implementation.