Text Segmentation with Intl.Segmenter (Word, Sentence, Grapheme)

Intl.Segmenter: Locale-Aware Text Segmentation

Intl.Segmenter splits text into meaningful segments (words, sentences, or grapheme clusters) according to locale-specific rules. This is particularly important for languages that do not use spaces between words.

Word Segmentation

// English: spaces separate words
const en = new Intl.Segmenter('en', { granularity: 'word' });
const words = [...en.segment('Hello world!')].filter(s => s.isWordLike);
// [{segment: "Hello"}, {segment: "world"}]

// Japanese: no spaces between words
const ja = new Intl.Segmenter('ja', { granularity: 'word' });
const jaWords = [...ja.segment('東京は日本の首都です')].filter(s => s.isWordLike);
// [{segment: "東京"}, {segment: "日本"}, {segment: "首都"}]

Why Spaces Are Not Enough

Many languages do not use spaces between words:

• Japanese: 私は学生です (watashi wa gakusei desu)
• Chinese: 我是学生 (wo shi xuesheng)
• Thai: ฉันเป็นนักเรียน (chan pen nak rian)
• Khmer, Lao, Myanmar: No word-boundary spaces

Sentence Segmentation

const seg = new Intl.Segmenter('en', { granularity: 'sentence' });
const text = 'Hello world. How are you? I am fine!';
const sentences = [...seg.segment(text)].map(s => s.segment);
// ["Hello world. ", "How are you? ", "I am fine!"]

Grapheme Cluster Segmentation

const seg = new Intl.Segmenter('en', { granularity: 'grapheme' });

// Emoji with modifiers count as one grapheme
const emoji = '👨‍👩‍👧‍👦'; // family emoji
[...seg.segment(emoji)].length;  // 1 (not 7!)

// Accented characters
const accented = 'é'; // e + combining accent = é
[...seg.segment(accented)].length;  // 1 (not 2!)

// Compare with string length
emoji.length;     // 11 (UTF-16 code units)
accented.length;  // 2 (UTF-16 code units)

Practical Uses

// Character counter that handles emoji correctly
function countGraphemes(text, locale = 'en') {
  const seg = new Intl.Segmenter(locale, { granularity: 'grapheme' });
  return [...seg.segment(text)].length;
}

countGraphemes('Hello 😀');  // 7 (not 8)
countGraphemes('👨‍👩‍👧‍👦😊');  // 2

// Word count for CJK text
function countWords(text, locale) {
  const seg = new Intl.Segmenter(locale, { granularity: 'word' });
  return [...seg.segment(text)].filter(s => s.isWordLike).length;
}

countWords('東京は日本の首都です', 'ja');  // 3

Browser Support

Intl.Segmenter is supported in Chrome 87+, Edge 87+, Safari 15.4+, and Firefox 125+. It is relatively newer than other Intl APIs.