UTF-8 Encoding and Unicode Normalization

UTF-8 and Normalization

UTF-8 is a variable-length encoding for Unicode. Since normalization changes the code points in a string, it directly affects the UTF-8 byte representation.

How Normalization Changes UTF-8 Bytes

The character é (e with acute accent):

NFC (precomposed):

U+00E9 → 0xC3 0xA9 (2 bytes)

NFD (decomposed):

U+0065 → 0x65 (1 byte)
U+0301 → 0xCC 0x81 (2 bytes)
Total: 3 bytes

Byte Size Comparison

Character	NFC Bytes	NFD Bytes	Difference
é	2	3	+50%
ñ	2	3	+50%
Å	2	3	+50%
が (ga)	3	6	+100%
가 (ga, Korean)	3	6	+100%

Storage Impact

For languages heavy in diacritics (French, Vietnamese, German) or CJK with combining marks (Korean Jamo), NFD can significantly increase storage requirements:

• French text: ~5-10% larger in NFD vs NFC
• Korean text in Jamo: up to 100% larger in NFD vs NFC
• ASCII-only text: identical in all forms

UTF-16 and Normalization

In UTF-16 (used by JavaScript, Java, and Windows):

• NFC é: 1 code unit (0x00E9)
• NFD é: 2 code units (0x0065, 0x0301)

This means JavaScript's .length property returns different values for NFC vs NFD representations of the same text.

Wire Format Considerations

When transmitting text over networks:

1. Normalize before transmission to ensure consistency
2. NFC is preferred (smaller size, wider compatibility)
3. HTTP headers should be ASCII-only (use percent-encoding for non-ASCII)
4. JSON allows UTF-8 directly or \uXXXX escape sequences

Best Practice

Always normalize text to NFC before storing or transmitting as UTF-8. This minimizes byte size and ensures compatibility with the widest range of systems.