Fixing Mojibake

Why Text Turns to Garbage and How to Fix It

By the ASCII World team

Mojibake (from Japanese "character" + "transform") is garbled text caused by decoding bytes with the wrong character encoding. The string "cafe" stored as UTF-8 and read as ISO 8859-1 becomes "cafÃ©" because the two-byte UTF-8 sequence 0xC3 0xA9 gets interpreted as two separate Latin-1 characters. Every developer hits this eventually, usually in a database migration, a CSV export, or an API response that looked fine in testing. The fix is always the same pattern: identify the actual byte encoding, identify the assumed encoding, and either re-decode or convert.

1. How Mojibake Happens

Text files do not store letters. They store bytes. The letter "e" (e with acute accent) occupies one byte in ISO 8859-1 (0xE9) and two bytes in UTF-8 (0xC3 0xA9). If a program writes UTF-8 bytes and the reader assumes ISO 8859-1, each byte gets decoded independently. 0xC3 becomes "A with tilde" and 0xA9 becomes "copyright sign." The two-character sequence "Ã©" appears instead of "e."

This is not corruption. Your bytes are intact. Metadata (which encoding to use for decoding) is wrong, missing, or ignored. Fix the metadata and the text renders correctly. The encoding comparison tool shows exactly where ISO 8859-1 and UTF-8 byte values diverge.

2. Recognizing Common Patterns

Mojibake follows predictable patterns based on which encodings are swapped. Learn to recognize these and you can diagnose the problem on sight:

You See	Expected	Cause
`Ã©`	e (e-acute)	UTF-8 read as ISO 8859-1
`Ã¼`	u (u-umlaut)	UTF-8 read as ISO 8859-1
`Ã±`	n (n-tilde)	UTF-8 read as ISO 8859-1
`â€œ` and `â€`	Left/right double quotes	Windows-1252 smart quotes read as UTF-8
`â€™`	Right single quote / apostrophe	Windows-1252 read as UTF-8
`Ã¢â‚¬â€œ`	En dash	Double-encoded UTF-8 (UTF-8 bytes stored as UTF-8 again)
`?` or `�` (replacement char)	Any non-ASCII character	Bytes outside the target encoding's range

The pattern "A followed by a symbol" (Ã©, Ã¼, Ã±, Ã§) is the signature of UTF-8 decoded as Latin-1. Every 0xC3 byte (the UTF-8 lead byte for U+00C0-U+00FF) becomes "A with tilde" in Latin-1 because ISO 8859-1 position 0xC3 is that character. Use the hex to ASCII converter to check specific byte values.

3. Database Mojibake

The most common and most painful variant. MySQL databases set to latin1 (the default before MySQL 8.0) store UTF-8 bytes without converting them. Bytes survive, but the database thinks they are Latin-1. Three scenarios:

Scenario 1: latin1 column, UTF-8 connection

Application sends UTF-8 bytes. MySQL's latin1 connection charset passes them through without conversion. Bytes land in the column unchanged, but the column metadata says "latin1." Reading with a UTF-8 connection triggers conversion that double-encodes the already-UTF-8 bytes. The letter e (2 UTF-8 bytes) becomes 4 bytes.

Scenario 2: Double encoding already happened

Detect it by checking hex values directly:

SELECT HEX(name) FROM users WHERE id = 42;
-- If you see C383C2A9, the e is double-encoded
-- Single UTF-8 would be C3A9
-- The bytes C3A9 were treated as two Latin-1 chars,
-- each then re-encoded to UTF-8: C3 -> C383, A9 -> C2A9

Fix for MySQL

-- Step 1: Fix the column type without re-encoding bytes
ALTER TABLE users MODIFY name VARBINARY(255);
ALTER TABLE users MODIFY name VARCHAR(255) CHARACTER SET utf8mb4;

-- Step 2: Fix the connection and database defaults
ALTER DATABASE mydb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

-- Step 3: Verify
SELECT name, HEX(name) FROM users WHERE id = 42;
-- Should show C3A9 for e, not C383C2A9

Going through VARBINARY strips the charset metadata without touching the raw bytes. Going straight from latin1 to utf8mb4 would re-encode the bytes, making the problem worse. See the SQL article for the ASCII() and CHAR() functions useful for debugging these issues.

4. Email and HTTP Mojibake

Email messages declare their encoding in the Content-Type header: Content-Type: text/plain; charset=UTF-8. When this header is missing or wrong, the email client guesses. Outlook defaults to the system locale's encoding. Thunderbird defaults to ISO 8859-1 for messages without a charset declaration. Gmail defaults to UTF-8 but falls back to auto-detection.

HTTP has the same mechanism. The response header Content-Type: text/html; charset=utf-8 tells the browser which encoding to use. If the header says charset=iso-8859-1 but the body contains UTF-8 bytes, accented characters break. An HTML <meta charset="utf-8"> tag in the <head> acts as a fallback when the HTTP header is absent.

Fix email mojibake by viewing the raw message source (Ctrl+U in Thunderbird, "Show original" in Gmail) and checking the Content-Type header against the actual byte content. If the bytes are valid UTF-8 but the header says latin1, the sender's mail client is misconfigured.

5. Terminal and Shell Mojibake

Terminal emulators have their own encoding setting independent of the shell and the file system. On macOS, Terminal.app defaults to UTF-8. On older Linux systems, the locale might be en_US.ISO-8859-1 instead of en_US.UTF-8. Check with locale and look for LC_ALL or LANG.

# Check current locale
locale
# Fix: add to ~/.bashrc or ~/.zshrc
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8

SSH sessions inherit the client's locale. If your local machine uses UTF-8 but the remote server uses Latin-1, filenames with accented characters display as mojibake. Set SendEnv LANG LC_* in your SSH client config and AcceptEnv LANG LC_* on the server to propagate locale settings.

Piping binary data through commands that assume text encoding can also produce mojibake. Commands like sort, grep, and awk use the current locale (based on the system's ASCII-derived sort order)'s encoding. If a file contains raw bytes that are not valid in the locale's encoding, these tools may silently drop or replace characters. Use LC_ALL=C prefix to force byte-level processing: LC_ALL=C sort file.txt.

6. Fix Recipes

Python: repair double-encoded text

# The string was UTF-8 bytes misread as Latin-1, then encoded to UTF-8 again
broken = "cafÃ©"  # This is what you see
fixed = broken.encode('latin-1').decode('utf-8')
# Result: "cafe" (with accent)

# For triple-encoding (yes, it happens):
fixed = broken.encode('latin-1').decode('utf-8')
fixed = fixed.encode('latin-1').decode('utf-8')

The pattern is always: encode back to bytes using the wrong encoding, then decode using the right one. See the Python article for more on encode() and decode() methods.

iconv: convert file encoding

# Convert from ISO 8859-1 to UTF-8
iconv -f ISO-8859-1 -t UTF-8 input.txt > output.txt

# Convert from Windows-1252 to UTF-8
iconv -f WINDOWS-1252 -t UTF-8 input.txt > output.txt

# List available encodings
iconv -l

Vim: re-read file with correct encoding

:e ++enc=utf-8
:e ++enc=latin1
:set fileencoding=utf-8
:w

PostgreSQL: check and fix column encoding

-- Check database encoding
SHOW server_encoding;

-- Convert a column's encoding
UPDATE users SET name = convert_from(
  convert_to(name, 'LATIN1'), 'UTF8'
) WHERE name LIKE '%Ã%';

7. Prevention

Every system in the chain must agree on UTF-8. There is no second-best option.

Databases: Set the default character set to utf8mb4 (MySQL) or ensure UTF8 encoding (PostgreSQL). Check connection charset too, not just the column/table level.
Files: Save as UTF-8 without BOM. Configure your editor's default encoding. Check the keyboard reference for which characters fall within standard ASCII. Git's .gitattributes can enforce this: * text=auto working-tree-encoding=UTF-8.
HTTP: Always send Content-Type: text/html; charset=utf-8. Add <meta charset="utf-8"> as the first element in <head>.
Email: Ensure your mail library sets charset=UTF-8 in Content-Type headers for every message.
APIs: JSON is UTF-8 by spec (RFC 8259). Do not accept or send JSON in other encodings.
Filenames: macOS uses NFD normalization for filenames. Linux uses whatever bytes you give it. Windows uses UTF-16. Cross-platform file sharing of filenames with accented characters is still a minefield.

Bookmark the encoding comparison tool for quick lookups when debugging. The ASCII chart is also useful for verifying standard code points. Browse the character sets reference for all supported encodings with their language coverage. And the text to ASCII converter shows the raw code point values behind any text string, which is the fastest way to verify what bytes you are actually dealing with.