Hidden Characters in Text Files: Line Endings, Whitespace and Zero-Width Characters

Some of the most frustrating bugs involve characters you cannot see. A script fails with a strange error, two strings that look identical do not match, or a diff shows every line changed even though nothing seems different. Invisible characters are usually the cause. Knowing the common culprits makes these problems quick to diagnose.

Line endings: LF, CRLF and CR

Different systems mark the end of a line differently. Unix-like systems, including Linux and macOS, use a line feed (LF, \n). Windows traditionally uses carriage return plus line feed (CRLF, \r\n). Very old Mac systems used CR alone.

Mixed or unexpected line endings cause problems such as:

• shell scripts failing with errors like bad interpreter because the first line ends in an invisible \r;
• diffs showing every line as changed after a file was saved on another system;
• CSV parsers producing values with a trailing carriage return.

Fix: choose one convention per project. Git can normalize line endings via a .gitattributes file, and editors can show and convert line endings.

Tabs, spaces and trailing whitespace

Tabs and spaces look similar, but some languages and formats care. Python treats inconsistent indentation as an error, YAML forbids tabs for indentation and Makefiles require tabs before commands. Trailing spaces at the end of lines create noisy diffs and, in Markdown, two trailing spaces create a line break.

Fix: configure your editor to show whitespace and trim trailing spaces, and use an .editorconfig file to share settings across a team.

Non-breaking and other special spaces

Unicode has many space characters besides the ordinary space (U+0020). The non-breaking space (U+00A0) is common in text copied from web pages and word processors. Others include the narrow no-break space and various fixed-width spaces. They look normal but are not matched by code that only looks for U+0020, which breaks splitting, trimming and number parsing.

Fix: normalize whitespace on input, for example by replacing all Unicode space separators with a regular space where appropriate.

Zero-width characters

Some characters have no width at all:

• Zero-width space (U+200B): marks a possible line break without showing a space.
• Zero-width joiner (U+200D) and non-joiner (U+200C): control how characters combine; the joiner is essential in some emoji sequences and scripts.
• Word joiner (U+2060) and the deprecated use of U+FEFF as a zero-width no-break space.

Copied text can carry these into identifiers, passwords, URLs or configuration keys, causing values that look equal to compare as different. Do not strip them blindly from natural-language text, because joiners are meaningful in some languages and emoji.

The byte order mark

The BOM (U+FEFF) at the start of a file signals the encoding. In UTF-8 it is optional and often unwanted: it can appear as an invisible character before the first field of a CSV file or cause a parse error at the start of a JSON or script file. Save UTF-8 without a BOM unless a tool explicitly requires one.

Bidirectional control characters

Unicode includes characters that change the display direction of text, used for mixing right-to-left and left-to-right scripts. In source code they can make text display in a different order from how a compiler reads it. Many editors and code-hosting platforms now warn about them.

How to find hidden characters

1. Turn on "show whitespace" or "show invisibles" in your editor.
2. View the raw bytes with a hex viewer, for example xxd file | head.
3. Use cat -A on Unix-like systems to display line endings and tabs.
4. Print the code points of a suspicious string in your programming language.
5. Paste the text into a diff tool that highlights whitespace differences.

Key takeaways

• Agree on line endings, indentation and encoding per project and enforce them.
• Normalize whitespace in input that becomes identifiers or data.
• When equal-looking strings differ, inspect their code points.