UTF-8 BOM Explained: Why It Breaks Code & How to Fix It

The Mechanics of U+FEFF Across Character Encodings

Unicode character U+FEFF was originally defined as the zero-width no-break space in Unicode 1.0. Beyond its role as an invisible typographical character, U+FEFF serves as a byte order signal when binary text stream serialization occurs across different computer architectures.

Architectures handle multi-byte integers using either big-endian (most significant byte first) or little-endian (least significant byte first) byte ordering. When a system reads a 16-bit encoding like UTF-16, it must determine which byte in a two-byte sequence comes first. The byte sequence for U+FEFF in UTF-16 Big Endian is 0xFE 0xFF. If a little-endian system reads this sequence without byte-swapping, it sees 0xFF 0xFE, which corresponds to the non-character code point U+FFFE. This signature instantly tells the parser to swap byte pairs for the remainder of the file.

The table below summarizes how U+FEFF manifests across different encoding schemes:

Encoding	Byte Sequence (Hexadecimal)	Endianness Indicator Role
UTF-16 Big Endian	FE FF	Explicitly indicates Big Endian ordering.
UTF-16 Little Endian	FF FE	Explicitly indicates Little Endian ordering.
UTF-32 Big Endian	00 00 FE FF	Explicitly indicates 32-bit Big Endian ordering.
UTF-32 Little Endian	FF FE 00 00	Explicitly indicates 32-bit Little Endian ordering.
UTF-8	EF BB BF	None. Used solely as an encoding signature.

Why UTF-8 Does Not Require Endianness Signals

UTF-8 serializes text into a sequence of single 8-bit bytes (octets). Because the basic unit of storage in UTF-8 is a single byte, byte ordering within a unit does not exist. A single byte has no endianness.

In UTF-8, multi-byte sequences for higher code points are built using fixed bit patterns where the leading byte indicates the total byte length of the character, and subsequent continuation bytes begin with the binary prefix 10. As detailed in our guide on how UTF-8 works, the reading order is strictly left-to-right at the byte level regardless of CPU architecture.

The Unicode Standard (Section 2.6) explicitly states that a BOM is neither required nor recommended for UTF-8 streams. The standard permits the UTF-8 BOM only as an optional signature to identify a stream as UTF-8 data, but warns that it disrupts software expecting plain text or specific byte formats like ASCII.

How the UTF-8 BOM Causes Software Failures

Although the Unicode Standard allows the 0xEF 0xBB 0xBF prefix, standard Unix utilities, compilers, interpreters, and web protocols expect binary streams to begin with specific operational bytes. When an unexpected BOM precedes those bytes, parsing fails.

1. Broken Shell Scripts and the Shebang Line

In Unix-like operating systems, an executable script begins with a shebang sequence: #!/bin/bash or #!/usr/bin/env python3. The operating system kernel reads the first two bytes of the file looking for the magic number 0x23 0x21 (the ASCII characters # and !).

If a shell script is saved in UTF-8 with a BOM, the file begins with hexadecimal bytes 0xEF 0xBB 0xBF 0x23 0x21. The kernel fails to match the shebang signature and attempts to execute the script under the default shell, or fails immediately with an error such as exec format error or command not found: #!/bin/bash.

2. PHP Header Injection and Cookie Errors

PHP interprets any content outside of <?php ... ?> tags as raw text to be sent directly to the HTTP output stream. If a PHP source file contains a UTF-8 BOM, the interpreter outputs those three bytes before processing the opening PHP tag.

This early output triggers the web server to send HTTP response headers immediately. If the script later attempts to call header(), setcookie(), or session_start(), PHP produces a fatal error or warning:

Cannot modify header information - headers already sent by (output started at /path/to/script.php:1)

This issue frequently manifests as strange layout bugs or silent session failures, resulting in difficult debugging scenarios described in our article on fixing mojibake and encoding bugs.

3. JSON Parsing Failures

The official JSON specification (RFC 8259, Section 8.1) explicitly forbids a BOM in JSON text. It states that implementations MUST NOT add a byte order mark to the start of a JSON text, and conforming parsers are not required to strip it.

When a JSON parser encounters 0xEF 0xBB 0xBF before the opening brace { or bracket [, it throws a syntax error. Node.js JSON.parse(), Python's json.loads(), and Go's json.Unmarshal will report invalid token errors at line 1, column 1.

4. Source Code Compilation and Configuration Files

Compilers for C, C++, Java, and Go treat the UTF-8 BOM differently based on their version and flag settings:

• GCC / Clang: Modern versions generally ignore a BOM at the start of C/C++ source files, but older versions flag it as an unknown character error.
• Go: The Go compiler (gc) strictly disallows a BOM in .go files, throwing a syntax error: hidden sequence UTF-8 BOM during build.
• Windows Configuration Files: Tools reading .ini, .env, or cross-platform CSV files may include the BOM bytes as part of the first key name. For example, a key named DB_HOST becomes \ufeffDB_HOST in Python or Node.js, causing environmental variable lookup failures.

Why Windows Editors Insert UTF-8 BOMs

Historically, Microsoft Windows used double-byte character sets (MBCS) and Windows code pages (such as Windows-1252) alongside UTF-16. To distinguish plain ANSI text files from UTF-8 files without reliance on MIME types or external metadata, Windows Notepad added the 0xEF 0xBB 0xBF sequence whenever a user saved a file as UTF-8.

This behavior persisted for decades until Windows 10 (version 1903), when Microsoft changed Notepad's default saving format to "UTF-8" (without BOM) and created a separate selection named "UTF-8 with BOM". However, legacy Windows applications, SQL Server Management Studio scripts, and certain PowerShell versions (such as PowerShell 5.1's Out-File -Encoding utf8) still insert the UTF-8 BOM by default.

Detecting and Stripping the UTF-8 BOM

Because the UTF-8 BOM is invisible in standard text editors, diagnosing its presence requires byte-level inspection tools.

Inspection Tools

On Linux or macOS terminal environments, the file command reveals whether a file contains a BOM:

$ file configuration.json
configuration.json: UTF-8 Unicode (with BOM) text

To inspect the raw hex bytes directly, use hexdump or xxd:

$ hexdump -C script.sh | head -n 1
00000000  ef bb bf 23 21 2f 62 69  6e 2f 62 61 73 68 0a     |...#!/bin/bash.|

The leading ef bb bf represents the BOM.

Removing BOMs from the Command Line

You can remove the BOM from a single file using sed:

# Linux (GNU sed)
sed -i '1s/^\xEF\xBB\xBF//' script.sh

# macOS (BSD sed)
sed -i '' '1s/^\xEF\xBB\xBF//' script.sh

Alternatively, the dos2unix utility provides a dedicated option to strip byte order marks across entire project directories:

dos2unix --remove-bom *.txt

For large-scale project cleanups, finding all BOM-impacted files with grep and stripping them with perl works efficiently across platforms:

# Find files containing UTF-8 BOM
grep -rlP '^\xEF\xBB\xBF' .

# Strip BOM from found files
find . -type f -name "*.py" -exec perl -pi -e 's/^\xEF\xBB\xBF//' {} +

Handling BOMs in Programming Languages

When reading external files programmatically, handle potential BOM sequences at the application level:

• Python: Use the utf-8-sig encoding codec when reading text. Python will automatically detect and strip the BOM if present, but parse normal UTF-8 normally if absent.

  with open('data.csv', 'r', encoding='utf-8-sig') as f:
      content = f.read()

• Node.js: Remove the BOM using a string replacement or buffer check before passing string content to JSON parsers:

  if (data.startsWith('\uFEFF')) {
      data = data.slice(1);
  }

• C# / .NET: Use new StreamReader(path, Encoding.UTF8). The default UTF8Encoding class in .NET detects and consumes the BOM automatically without returning it in the output string.

Preventing BOMs in Development Workflows

Configuring development tools ensures BOMs are not introduced into source repositories accidentally.

• Visual Studio Code: Set "files.encoding": "utf8" in settings. Avoid choosing utf8bom unless explicitly targeting legacy Windows platforms.
• Git Configuration: Add a .gitattributes rule or pre-commit hook to catch files saved with byte order marks before they reach the remote repository.
• PowerShell 7+: Upgrade scripts from Windows PowerShell 5.1 to PowerShell 7, which defaults to BOM-less UTF-8 across all file-writing cmdlets.