Unicode & string internals
Learn why an emoji has length 2, how JavaScript strings store UTF-16 code units, how to count code points and graphemes, and how normalization and well-formed text keep user input safe.
- 01Name the three countsTell code units, code points, and graphemes apart when you read what a string method gives back.
- 02Pick the right toolChoose
codePointAt,fromCodePoint,normalize, orIntl.Segmenterfor each job. - 03Keep text safeShorten text without breaking emoji, compare text after normalizing it, and repair or refuse broken strings.
Why an emoji has length 2
In JavaScript, a string is text inside quotes. You met strings in the Strings lesson. That lesson showed one odd result: "😀".length is 2. A person sees one smiling face. So why 2?
The answer is in how JavaScript stores text. It doesn't store "characters". It stores small pieces called UTF-16 code units. Each code unit is a 16-bit number: a whole number from 0 to 65,535. Most letters need one code unit. This emoji needs two.
length counts the pieces, so it gives 2.People use the word "character" for three different things. This lesson keeps them apart:
- Code units: the small pieces JavaScript stores and counts.
- Code points: the numbers Unicode gives to each letter, mark, or emoji.
- Graphemes: what a person sees as one character on the screen.
Unicode is a big list that gives every letter, mark, and emoji its own number. That number is called a code point. JavaScript stores each code point as one or two UTF-16 code units. People care about graphemes: what they see as one character.
Picture a big shop catalog. Every product in it has its own number. To order, you write that number on a paper form. The form has boxes of a fixed size. A short number fits in one box. A long number needs two boxes, side by side. Unicode is the catalog. UTF-16 is the form that JavaScript strings use.
- In real life: The catalog gives every product a number
- In JavaScript: Unicode gives every character a code point, like U+1F600 for 😀
- In real life: Most numbers fit in one box on the form
- In JavaScript: Most code points fit in one UTF-16 code unit
- In real life: A long number needs two boxes
- In JavaScript: Emoji and some rare characters need two code units, called a surrogate pair
- In real life: The customer sees one product
- In JavaScript: A person sees one character, even when JavaScript stores two code units
Two later lessons go deeper. Text encoding & Base64 covers UTF-8 bytes and TextEncoder. Strings inside the engine covers how a JavaScript engine keeps strings in memory. This lesson stays with the string tools you use in everyday code.
UTF-16 code units: the pieces JavaScript counts
INTERACTIVEA code unit is one 16-bit piece of a string. Many tools you know from String methods work in code units:
lengthcounts code units.- An index in square brackets, like
text[0], reads one code unit. charCodeAtgives the number stored in one code unit.slicecuts the string between code units.
That is useful when you need the exact stored pieces. But it is not always the same as what a person sees.
const name = "Asha";console.log(name.length);console.log(name[0]);Here the results are what you would expect. Line 1 stores Asha. Line 2 prints its length, 4, because it has four code units. Line 3 prints index 0, which is A.
Plain English letters (called ASCII) each fit in one code unit. So with them, you can't see the difference between code units and characters.
Now try the X-ray playground below. It counts the same text in three ways. Try these samples: héllo, 😀, 👍🏽, a family emoji, the two kinds of é, and a flag. Watch the three counts disagree.
const text = "😀";text.length;text.split("");[...text];Array.from(new Intl.Segmenter(undefined, { granularity: "grapheme" }).segment(text));split("")["\ud83d","\ude00"]Code units
0 0xD83D
1 0xDE00
Code points
0 😀 U+1F600
Graphemes
0 😀
Your browser counts 2 code units, 1 code points, and 1 grapheme. Checking for Intl.Segmenter.
Intl.Segmenter, the playground counts code points instead.Look closely at the split("") row. split("") cuts the text between code units, so it can tear an emoji into two halves.
Spread syntax, [...text], works differently. It walks through the text one code point at a time, so it keeps both halves of an emoji together. The for…of & for…in lesson walks through strings the same way.
length counts 4 code units. Spread, [...text], counts 2 code points. Intl.Segmenter counts 1 grapheme: what a person sees.The table below names the three counts. Read each row as: the unit, how JavaScript counts it, and an example.
| The unit | How JavaScript counts it | Example |
|---|---|---|
| Code unit | One 16-bit UTF-16 piece. length, indexes, charCodeAt, and slice count these. | "😀".length is 2. |
| Code point | One Unicode number. for…of, spread, codePointAt, and fromCodePoint work with these. | [..."😀"].length is 1. |
| Grapheme | One character as a person sees it. Intl.Segmenter with granularity: "grapheme" splits text into these. | 👍🏽 is one grapheme but two code points. |
Code points & surrogate pairs
A code point is the number Unicode gives to a letter, mark, or emoji. Code points are written as U+ and a hex number. Hex (short for hexadecimal) writes numbers with the digits 0 to 9 and the letters A to F. For example:
- The smiling face
😀is U+1F600. - The letter
Ais U+0041.
One code unit can hold code points up to U+FFFF. Bigger code points need two code units. Those two code units are called a surrogate pair:
- The first one is the high surrogate.
- The second one is the low surrogate.
Only the two together stand for the code point.
Some concerts print each ticket in two halves. The ticket only works when both halves stay together. A surrogate pair works the same way. JavaScript can keep one half in a string, but one half on its own is broken text.
- In real life: Some concert tickets come in a left half and a right half
- In JavaScript: One code point can be stored as a high surrogate and a low surrogate
- In real life: One half alone won't get you in
- In JavaScript: One surrogate alone, called a lone surrogate, is not a real character
- In real life: The scanner reads both halves together
- In JavaScript:
codePointAt(0)reads both code units when index 0 is the high surrogate
Now step through a real emoji. Every number below comes from JavaScript itself. A number that starts with 0x is written in hex:
55357is0xD83D, the first code unit.128512is0x1F600, the code point.
Guess each result before you step. This replay shows the difference between what a person sees and what JavaScript stores.
script
console.log(smile.length);console.log(smile.charCodeAt(0));console.log(smile.codePointAt(0));console.log(String.fromCodePoint(128512));console.log(String.fromCharCode(0xD83D, 0xDE00));console.log(smile.slice(0, 1));codePointAt, fromCodePoint & fromCharCode
STEP THROUGHThere are two pairs of tools. They differ in what they work with:
charCodeAtandString.fromCharCodework with UTF-16 code units. "Char code" is the older name for the number in one code unit.codePointAtandString.fromCodePointwork with Unicode code points.
const smile = String.fromCodePoint(0x1F600);console.log(smile);console.log(smile === String.fromCharCode(0xD83D, 0xDE00));Line 1 builds 😀 from its code point, 0x1F600. Line 2 prints it. Line 3 builds it again from its two code units and compares the two. They match, so it prints true.
When you have a Unicode number, use fromCodePoint. The older fromCharCode only gives the same emoji here because line 3 passes both code units.
codePointAt(index) reads both halves of a pair only when index points at the high surrogate, the first half. If index points at the low surrogate, you get just that one code unit. A for…of loop and spread never stop in the middle of a pair, so they don't have this problem.
Read each row of this table as: the tool, what it works with, and when to use it.
| Tool | What it works with | Use it when |
|---|---|---|
charCodeAt(index) | One code unit | You need the exact UTF-16 code unit at an index. |
codePointAt(index) | One code point | You want the Unicode number that starts at this index. |
String.fromCharCode(...) | Code units | You already have UTF-16 code units, including both halves of a pair. |
String.fromCodePoint(...) | Code points | You have Unicode numbers, such as 0x1F600. |
In the sorter below, pick the unit each tool counts. If you get one wrong, the card tells you why.
str.lengthstr.charCodeAt(0)str.slice(0, 1)[...str]str.codePointAt(0)String.fromCodePoint(0x1F600)Intl.Segmenter(..., { granularity: "grapheme" })- A limit on visible characters in a profile name
Sort each tool by the unit it counts or keeps whole.
normalize(): one spelling for the same word
STEP THROUGHSometimes Unicode has two correct ways to store the same letter. Take é:
- It can be one ready-made code point, U+00E9.
- Or it can be the letter
efollowed by a combining accent, U+0301. A combining mark is a code point that sits on top of the letter before it.
Both look the same in most fonts. But === compares the stored code units, not the look. So the two strings are not equal.
é, so === says false at first. After normalize(), both use the same one-code-point spelling, so === says true.One list says "A. Lovelace". Another list says "Ada Lovelace". A person can tell these are the same name. A computer needs a rule. Normalization is that rule for the different Unicode spellings of the same text.
- In real life: Two people write the same name in different ways
- In JavaScript: Two strings look the same but store different code points
- In real life: A form asks everyone to use the official spelling
- In JavaScript:
normalize()rewrites text into one standard spelling - In real life: Now the names can be compared fairly
- In JavaScript: Normalized strings can be compared, searched, and checked for repeats
Step through the two spellings of é below.
The two strings look the same. Step through to see why === says false before normalize() and true after it.
script
const decomposed = "é";console.log(composed === decomposed);console.log(composed.length);console.log(decomposed.length);console.log(composed.normalize() === decomposed.normalize());console.log(composed.normalize("NFD").length);A good rule: normalize text before you do any of these:
- compare what a user typed with something else,
- save it as a key, such as a key in a database,
- remove names that appear twice,
- search text where accents matter.
normalize() uses a form called NFC by default. NFC joins a letter and its accent into one ready-made code point where it can. Another form, NFD, splits them apart into a letter plus combining marks. NFD helps when you want to remove accents. For everyday comparing, use NFC.
const typed = "Cafe\u0301";const key = typed.normalize();console.log(key);console.log(key === "Café".normalize());Line 1 holds what the user typed: Cafe plus a combining accent. Line 2 makes a normalized copy called key. Line 3 prints it. Line 4 compares it with a normalized "Café" and prints true.
Keep typed to show on the screen. Use key only for comparing. Normalizing doesn't make text lowercase, and it doesn't remove accents. It only picks one standard spelling.
Graphemes with Intl.Segmenter
INTERACTIVEA grapheme is what a person sees as one character. One grapheme can be made of several code points stuck together. For example:
- a thumbs-up plus a skin-tone code point,
👍🏽, - a family of three people, joined by invisible code points called zero-width joiners,
- a flag, made of two letter-like code points called regional indicators,
- the letter
eplus a combining accent.
const text = "👍🏽";console.log([...text].length);console.log(Array.from(new Intl.Segmenter(undefined, { granularity: "grapheme" }).segment(text)).length);Line 2 uses spread. It finds two code points in 👍🏽: the thumbs-up and its skin tone. Line 3 uses Intl.Segmenter, a tool that splits text into graphemes. It finds one grapheme. That is the number a limit on visible characters needs.
Line 3 has several parts:
new Intl.Segmenter(undefined, { granularity: "grapheme" })makes the splitter.undefinedmeans "use the default language".granularity: "grapheme"means "split into graphemes"..segment(text)gives back the pieces.Array.from(...)turns the pieces into an array, so.lengthcan count them.
Intl.Segmenter works in Node 16 and newer, and in modern browsers. This page checks that it exists before using it. If you must support an older browser, use a tested library that splits graphemes. Or count code points instead, and accept that the count is less exact.
A common bug is shortening text with slice. slice cuts between code units. So it can leave half an emoji, which shows as a broken box. Or it can cut the skin tone off a thumbs-up. The safe way is to split the text into graphemes first, then keep the first few.
"👍🏽 launch". slice cuts by code unit and leaves a broken half. Cutting by grapheme keeps the whole thumbs-up.const text = "Hi 👍🏽";const parts = Array.from(new Intl.Segmenter(undefined, { granularity: "grapheme" }).segment(text), part => part.segment);console.log(parts.slice(0, 4).join(""));Line 2 splits "Hi 👍🏽" into graphemes and keeps the text of each piece. Line 3 keeps the first four pieces and joins them. They are H, i, a space, and the whole 👍🏽. The original text doesn't change, so you can still use it.
In the playground below, change the text and the limit. Compare the slice result with the grapheme result.
function firstGraphemes(text, limit) { const segmenter = new Intl.Segmenter(undefined, { granularity: "grapheme" }); return Array.from(segmenter.segment(text), part => part.segment) .slice(0, limit) .join("");} console.log("👍🏽 launch".slice(0, 2));console.log(firstGraphemes("👍🏽 launch", 1));slice cuts between code units and gives "\ud83d". Cutting between graphemes gives "👍🏽", with every character whole.
slice, which cuts between UTF-16 code units. The safe line cuts between graphemes.isWellFormed & toWellFormed
STEP THROUGHA string can hold a lone surrogate: one half of a surrogate pair, with the other half missing. A string like that is called ill-formed. A string with no lone surrogates is well-formed.
Two newer methods help. They work in Node 20 and newer, and in recent browsers:
isWellFormed()returnsfalseif the string has a lone surrogate.toWellFormed()returns a copy where each lone surrogate is replaced with U+FFFD, the replacement character�.
Why does this matter? Tools that send text out of JavaScript often need valid Unicode. For example, encodeURIComponent turns text into a safe form for a web address. But encodeURIComponent("\uD83D") throws a URIError. So fix or refuse broken text before you send it to a server or put it in a URL.
const broken = String.fromCharCode(0xD83D);console.log(broken.isWellFormed());console.log(encodeURIComponent(broken.toWellFormed()));Line 1 builds a broken string: only the first half of 😀. Line 2 asks isWellFormed(), which prints false. Line 3 repairs the string, then encodes it. The result is %EF%BF%BD, which is how a URL writes U+FFFD.
Replacing is a choice you make. It doesn't bring back the missing half. Step through the same broken string below. The last step shows the URIError.
Step through a string that is broken on purpose. isWellFormed and toWellFormed work in Node 20 and newer and in recent browsers; this page checks for them first.
script
console.log(broken.isWellFormed());console.log(broken.toWellFormed());try { encodeURIComponent(broken);} catch (error) { console.log(error.name);}This page checks for these methods before using them. If a browser doesn't have them, you can check surrogate pairs with a small helper function or a well-tested library. Check for a feature instead of assuming every visitor has the newest browser.
Where you’ll use this
These details matter whenever text moves from one place to another. For example: a profile name, a search box, a URL slug (the readable part of a web address), a database key, or a character counter. Most bugs come from using the wrong count for the job.
Read each row of this table as: the job, what to use, and why.
| The job | What to use | Why |
|---|---|---|
| Work with an older tool or format that uses UTF-16 | Code units | That tool is built around UTF-16 code units. |
| Reverse simple emoji text in a practice puzzle | Code points, with spread | Spread keeps surrogate pairs together. It doesn't keep every grapheme together. |
| Limit a display name to 20 visible characters | Graphemes | People expect 👍🏽 and 🇯🇵 to count as one character each. |
| Compare what a user typed, for search or to find repeats | normalize() first | Two spellings like é and e plus an accent should match. |
| Put text into a URL | A well-formed check | A lone surrogate makes encodeURIComponent throw. |
The next lesson, Comparing & sorting text, builds on this. It normalizes text before comparing. It also uses localeCompare and Intl.Collator to sort words in the order a person's language expects.
Common misconceptions
“length counts characters.”
It counts UTF-16 code units. For plain English letters, that is the same number. For emoji, accents made from combining marks, and many rare characters, it is not.
“charCodeAt gives the Unicode character number.”
It gives the number in one code unit. If the text might hold surrogate pairs and you want the code point, use codePointAt.
“Spread always counts the characters people see.”
Spread counts code points. It keeps 😀 together. But 👍🏽 is still two code points, even though a person sees one grapheme.
“If two strings look the same, === is true.”
=== compares the stored code units. If two spellings should count as the same text, normalize both first.
“JavaScript strings are UTF-8.”
JavaScript counts and indexes strings in UTF-16 code units. UTF-8 is a different way to store text, as bytes. It matters when you save text to a file or send it over a network. The later Text encoding & Base64 lesson covers it.
Practice: Unicode strings
5 EXERCISESGuess all four outputs before you run the code.
console.log("😀".length);
console.log([..."😀"].length);
console.log("👍🏽".length);
console.log([..."👍🏽"].length);The outputs are 2, 1, 4, and 2. The thumbs-up with a skin tone is two code points, stored as four UTF-16 code units.
Reverse Hi 😀 without splitting the emoji.
const text = "Hi 😀";
console.log([...text].reverse().join(""));const text = "Hi 😀";
console.log([...text].reverse().join(""));[...text] walks the text one code point at a time. For this string that is enough: the emoji stays whole, and the output is 😀 iH. Reversing text with families or skin tones is harder, because each of those is several code points.
What do the two comparisons print?
const typed = "Café";
const saved = "Café";
console.log(typed === saved);
console.log(typed.normalize() === saved.normalize());The stored strings are different, so the first output is false. After normalize(), both use the same spelling, so the second output is true.
How many visible characters should a profile-name counter show for 👍🏽👍?
const segmenter = new Intl.Segmenter(undefined, { granularity: "grapheme" });
const text = "👍🏽👍";
console.log(Array.from(segmenter.segment(text)).length);The string 👍🏽👍 has two graphemes: 👍🏽 and 👍. It has more code units than that, but a character counter should show 2.
Show the first visible character of 👍🏽 launch without breaking it.
const segmenter = new Intl.Segmenter(undefined, { granularity: "grapheme" });
const text = "👍🏽 launch";
const safe = Array.from(segmenter.segment(text), part => part.segment).slice(0, 1).join("");
console.log(safe);const segmenter = new Intl.Segmenter(undefined, { granularity: "grapheme" });
const text = "👍🏽 launch";
const safe = Array.from(segmenter.segment(text), part => part.segment).slice(0, 1).join("");
console.log(safe);The code splits the text into graphemes, keeps the first one, and joins it back into a string. The result is the whole 👍🏽. It isn't a broken half, and it isn't a thumbs-up without its skin tone.
Quiz: check your understanding
7 QUESTIONSEach answer explains why it is right or wrong. Read the wrong ones too. Most Unicode mistakes come from using the wrong count.
Question 1 of 7What does a string's
lengthcount?Choose an answer to see the explanation.
Question 2 of 7What does the emoji length snippet print?
Read the code, then predictconsole.log("😀".length);Choose an answer to see the explanation.
Question 3 of 7What does the emoji code point snippet print?
Read the code, then predictconsole.log("😀".codePointAt(0));Choose an answer to see the explanation.
Question 4 of 7Why normalize what a user typed before comparing it?
Choose an answer to see the explanation.
Question 5 of 7What do the normalization comparisons print?
Read the code, then predictconsole.log("é" === "e\u0301"); console.log("é".normalize() === "e\u0301".normalize());Choose an answer to see the explanation.
Question 6 of 7Which tool should you use to count visible characters for a limit on screen?
Choose an answer to see the explanation.
Question 7 of 7What happens with a lone surrogate in
encodeURIComponent?Read the code, then predicttry { encodeURIComponent(String.fromCharCode(0xD83D)); } catch (error) { console.log(error.name); }Choose an answer to see the explanation.
Key takeaways
- JavaScript stores and counts strings as UTF-16 code units.
- A code point is Unicode's number for a character. Code points above U+FFFF take two code units, called a surrogate pair.
codePointAtandString.fromCodePointwork with code points.charCodeAtandString.fromCharCodework with code units.normalize()turns different spellings of the same text into one spelling, so they compare as equal.Intl.Segmentercounts graphemes. Use it for character limits and to shorten text safely.isWellFormedfinds lone surrogates, andtoWellFormedrepairs them. Use them before you encode text.
Remember the one-liner.
Code units are the stored pieces, code points are Unicode's numbers, and graphemes are what people see.
Up next: Comparing & sorting text. There, normalize() meets localeCompare, which sorts words in the order a person's language expects.