Text looks simple until it crosses a boundary: a file read from disk, a database column, an HTTP request, a CSV export. At each boundary, bytes are turned into characters using an encoding, and when two sides disagree, the result is garbled text or a subtle bug. Most problems come from a small set of recurring mistakes.
1. Assuming the default encoding
Many languages and libraries use a platform default encoding when you do not specify one, and that default differs between systems. Code that works on one machine produces errors on another. Fix: always pass an explicit encoding, normally UTF-8, when reading or writing text files and streams.
2. Mojibake from double decoding
Seeing "é" instead of "é" usually means UTF-8 bytes were decoded as Latin-1 or Windows-1252. If the wrong text was then saved again, the error becomes permanent. Fix: find the boundary where the wrong decoding happened rather than patching characters afterwards. Libraries exist that can repair common mojibake, but prevention is far more reliable.
3. Confusing Latin-1 and Windows-1252
The two are similar, but Windows-1252 assigns printable characters such as curly quotes and the euro sign to bytes that are control codes in Latin-1. Text labelled as one but written as the other shows odd symbols. Fix: when legacy data claims to be Latin-1, test decoding as Windows-1252, and convert to UTF-8 once at import.
4. The byte order mark (BOM)
Some editors write the bytes EF BB BF at the start of UTF-8 files. Parsers that do not expect them may treat the BOM as part of the first field, producing a mysterious invisible character in a CSV header or a JSON parse error. Fix: save UTF-8 without a BOM when possible, or use a decoder option that strips it (for example, Python's utf-8-sig).
5. Counting bytes instead of characters
In UTF-8, ASCII characters take one byte, but others take two to four. Truncating a string at a byte limit can cut a character in half and produce invalid data. Fix: know whether a limit is in bytes or characters, and truncate on character boundaries.
6. Surrogate pairs in UTF-16 strings
Languages such as JavaScript and Java store strings as UTF-16 code units. Characters outside the Basic Multilingual Plane, including many emoji, take two code units, so "😀".length is 2 in JavaScript. Slicing in the middle splits the pair. Fix: use code-point-aware functions, such as iterating with for...of or Array.from() in JavaScript.
7. Grapheme clusters
Even code points are not what users see as characters. A flag emoji, a family emoji or a letter with combining accents can consist of several code points that display as one symbol. Fix: for user-facing operations such as cursor movement or "max 20 characters" limits, use grapheme segmentation, for example Intl.Segmenter in JavaScript.
8. Unnormalized comparisons
"é" can be stored as one code point or as "e" plus a combining accent. The strings look identical but compare as different, breaking lookups, deduplication and logins. Fix: normalize to a consistent form, usually NFC, before comparing or storing identifiers.
9. Database and connection mismatches
A database column, table or connection can each have its own character set. MySQL's legacy utf8 character set stores at most three bytes per character and cannot hold many emoji, while utf8mb4 supports the full range. Fix: use a full UTF-8 character set for storage and set the connection encoding explicitly.
10. Missing charset in HTTP and HTML
If a server does not declare the encoding, browsers and clients guess, and guesses can be wrong. Fix: send Content-Type: text/html; charset=utf-8 (or the JSON equivalent) and include <meta charset="utf-8"> early in HTML documents.
A simple policy that prevents most bugs
- Decode bytes to text as early as possible, at the input boundary.
- Work with text internally, normalized where identity matters.
- Encode to UTF-8 as late as possible, at the output boundary.
- Declare the encoding explicitly everywhere it can be declared.
Key takeaways
- Most encoding bugs come from a mismatch at a boundary, not from the data.
- Be explicit: UTF-8 everywhere, declared in files, databases and headers.
- Know whether you are counting bytes, code units, code points or graphemes.