Text encoding & Base64

Strings are not bytes

JavaScript lets you write "café" as if it were one simple value, but the browser stores strings as UTF-16 code units. Files, network bodies, cryptographic hashes, images, and typed arrays use bytes. Text encoding is the bridge between those two worlds.

If Unicode terms feel fuzzy, review Unicode & strings. If byte arrays feel new, review the neighboring lesson on ArrayBuffer & typed arrays. Later, the Files & blobs lesson uses the same conversions when reading and writing user files.

UTF-8 is the web's default byte encoding for text. ASCII characters keep their single byte. Larger code points use leading-byte patterns that announce how many continuation bytes follow. The prefixes are why a decoder can read a stream without spaces between characters.

The next recording implements a tiny UTF-8 encoder for valid code points and proves it agrees with TextEncoder for "Aé€😀". The goal is not to replace the platform API; it is to make the byte patterns inspectable.

1function encodeCodePoint(codePoint) {Run2  if (codePoint <= 0x7F) return [codePoint];3  if (codePoint <= 0x7FF) {4    return [0xC0 | (codePoint >> 6),5            0x80 | (codePoint & 0x3F)];6  }7  if (codePoint <= 0xFFFF) {8    return [0xE0 | (codePoint >> 12),9            0x80 | ((codePoint >> 6) & 0x3F),10            0x80 | (codePoint & 0x3F)];11  }12  return [0xF0 | (codePoint >> 18),13          0x80 | ((codePoint >> 12) & 0x3F),14          0x80 | ((codePoint >> 6) & 0x3F),15          0x80 | (codePoint & 0x3F)];16}17 18function encodeUtf8(text) {19  const bytes = [];20  for (const character of text) {21    const codePoint = character.codePointAt(0);22    bytes.push(...encodeCodePoint(codePoint));23  }24  return new Uint8Array(bytes);25}26 27const text = "Aé€😀";28console.log([...encodeUtf8(text)].join(","));29console.log([...new TextEncoder().encode(text)].join(","));

Notice the emoji: JavaScript stores it as two UTF-16 code units, so "😀".length is 2. A for...of loop walks the full code point, and UTF-8 writes four bytes. That is why byte length, code-unit length, and what a person calls “one character” can all differ.

TextEncoder has one job: turn a string into UTF-8 bytes. It does not accept a label because it always writes UTF-8. encode() creates a new typed array; encodeInto() writes into an existing Uint8Array, which is useful when a parser or stream already owns a buffer.

1const encoder = new TextEncoder();2const bytes = encoder.encode("éuro");3console.log([...bytes].join(","));4 5const destination = new Uint8Array(5);6const result = encoder.encodeInto("éuro", destination);7console.log(result.read + " code units");8console.log(result.written + " bytes");9console.log([...destination].join(","));

TextDecoder goes from bytes back to strings. It accepts labels such as "utf-8", "utf-16le", and "windows-1252". The label must match the data source, not the page you wish you had. A byte like 0xE9 is a complete é in Windows-1252, but an invalid standalone byte in UTF-8.

1const win1252 = new Uint8Array([0x63, 0x61, 0x66, 0xE9]);2console.log(new TextDecoder("windows-1252").decode(win1252));3console.log(new TextDecoder("utf-8").decode(win1252));4 5const utf16le = new Uint8Array([0x48, 0x00, 0x69, 0x00]);6console.log(new TextDecoder("utf-16le").decode(utf16le));7 8const withBom = new Uint8Array([0xEF, 0xBB, 0xBF, 0x4F, 0x4B]);9console.log(new TextDecoder("utf-8").decode(withBom));10console.log(new TextDecoder("utf-8", { ignoreBOM: true }).decode(withBom).charCodeAt(0).toString(16));

Bad bytes have two honest outcomes. The default decoder replaces invalid sequences with U+FFFD, shown as �. A decoder created with { fatal: true } throws instead, which is safer for file formats and protocols where corruption should stop the parse.

1const bad = new Uint8Array([0xC3, 0x28]);2console.log(new TextDecoder("utf-8").decode(bad));3console.log(new TextDecoder("utf-8", { fatal: true }).decode(bad));

Streams add one more wrinkle: a chunk can end halfway through a multi-byte character. Pass { stream: true } until the final chunk so the decoder can keep the partial byte sequence instead of replacing it too early.

1const bytes = new Uint8Array([0x63, 0x61, 0x66, 0xC3, 0xA9]);Run2const first = bytes.slice(0, 4);3const second = bytes.slice(4);4 5console.log(new TextDecoder("utf-8").decode(first));6 7const decoder = new TextDecoder("utf-8", { fatal: true });8console.log(decoder.decode(first, { stream: true }));9console.log(decoder.decode(second));

Base64 groups three bytes into four 6-bit indexes. Those indexes pick characters from an alphabet: A-Z, a-z, 0-9, +, and /. Padding = fills the last four-character group when the byte count is not a multiple of three. Base64url swaps + for -, / for _, and often omits padding.

The old browser helpers btoa and atob do not accept arbitrary JavaScript text. They operate on binary strings, where each string character's code is one byte from 0 through 255. That is why this first example works.

1console.log(btoa("é"));2const binary = atob("6Q==");3console.log(binary.charCodeAt(0));

The Euro sign is outside that one-byte range, so btoa("€") throws an InvalidCharacterError. The fix is not to guess a code page; encode the text to UTF-8 bytes first.

1console.log(btoa("€"));

1function bytesToBinaryString(bytes) {2  return String.fromCharCode(...bytes);3}4function binaryStringToBytes(binary) {5  return Uint8Array.from(binary, (character) => character.charCodeAt(0));6}7 8const bytes = new TextEncoder().encode("€");9const base64 = btoa(bytesToBinaryString(bytes));10console.log([...bytes].join(","));11console.log(base64);12console.log(new TextDecoder().decode(binaryStringToBytes(atob(base64))));

The typed-array Base64 and hex helpers are the direct APIs learners always wanted: bytes in, encoded text out; encoded text in, bytes out. MDN marks toBase64, fromBase64, toHex, and fromHex as standard-track, Baseline 2025 APIs with Chrome 140, Firefox 133, Safari 18.2, and Node 25 support. The CI runtime for this site is Node 22, so examples feature-detect and keep fallbacks.

1const bytes = new Uint8Array([104, 105, 63]);2if (typeof bytes.toBase64 === "function" && typeof Uint8Array.fromBase64 === "function") {3  console.log(bytes.toBase64());4  console.log(bytes.toBase64({ alphabet: "base64url", omitPadding: true }));5  console.log([...Uint8Array.fromBase64("aGk_", { alphabet: "base64url" })].join(","));6  console.log([...Uint8Array.fromBase64("SGk", { lastChunkHandling: "loose" })].join(","));7} else {8  console.log("native Base64 helpers unavailable");9}10 11const token = new Uint8Array([222, 173, 190, 239]);12if (typeof token.toHex === "function" && typeof Uint8Array.fromHex === "function") {13  console.log(token.toHex());14  console.log([...Uint8Array.fromHex("deadbeef")].join("-"));15} else {16  console.log([...token].map((byte) => byte.toString(16).padStart(2, "0")).join(""));17}

The Base64 options are worth naming. { alphabet: 'base64url' } switches to URL-safe - and _. { omitPadding: true } drops trailing =. Uint8Array.fromBase64 also accepts lastChunkHandling; "loose" accepts unpadded final chunks, while stricter modes reject or stop before partial input.

Hex is simpler: one byte becomes exactly two base-16 digits. Until native toHex is available everywhere you run, this manual form is easy to audit.

1const bytes = new Uint8Array([0, 15, 16, 255]);2const hex = [...bytes]3  .map((byte) => byte.toString(16).padStart(2, "0"))4  .join("");5console.log(hex);6 7const roundTrip = Uint8Array.from(hex.match(/../g), (pair) => parseInt(pair, 16));8console.log([...roundTrip].join(","));

Use this workbench as a ruler. Type text and compare the UTF-16 code-unit count with code points, UTF-8 bytes, hex, Base64, and base64url. The visible source is intentionally small: encode once, then derive every byte-oriented representation from the same bytes.

1const text = "Aé€😀";2const bytes = new TextEncoder().encode(text);3const hex = [...bytes].map((byte) => byte.toString(16).padStart(2, "0")).join("");4const base64 = btoa(String.fromCharCode(...bytes));5 6console.log([...bytes].join(","));7console.log(hex);8console.log(base64);

JWTs demonstrate why “encoded” does not mean “private.” The middle part is base64url JSON. Decoding it is useful for display and debugging, but a security decision must verify the JWT signature on the server.

1function base64UrlToBytes(part) {2  const base64 = part3    .replace(/-/g, "+")4    .replace(/_/g, "/")5    .padEnd(Math.ceil(part.length / 4) * 4, "=");6  const binary = atob(base64);7  return Uint8Array.from(binary, (character) => character.charCodeAt(0));8}9 10const payloadPart = "eyJzdWIiOiIxMjMiLCJhZG1pbiI6ZmFsc2V9";11const json = new TextDecoder().decode(base64UrlToBytes(payloadPart));12console.log(json);13console.log(JSON.parse(json).admin);

Data URLs are the opposite direction: keep small file bytes inside a string. They are handy for tiny generated files and demos, but large Base64 data URLs bloat HTML and CSS.

1const bytes = new TextEncoder().encode("Hello");2const base64 = btoa(String.fromCharCode(...bytes));3console.log(`data:text/plain;base64,${base64}`);

Hashes use bytes too. The Web Crypto API is asynchronous, returns an ArrayBuffer, and leaves the display format to you. Hex is the usual digest display because it is stable, lowercase, and easy to compare.

1async function sha256Hex(message) {2  const bytes = new TextEncoder().encode(message);3  const digest = await crypto.subtle.digest("SHA-256", bytes);4  return [...new Uint8Array(digest)]5    .map((byte) => byte.toString(16).padStart(2, "0"))6    .join("");7}8 9sha256Hex("hello").then(console.log);

1for (const length of [1, 2, 3, 4, 12]) {2  const base64Length = Math.ceil(length / 3) * 4;3  console.log(length + " bytes -> " + base64Length + " Base64 chars");4}

Common misconceptions

• “A JavaScript string is already UTF-8.” It is UTF-16 code units. UTF-8 appears after you encode it.
• “btoa encodes Unicode text.” It encodes binary strings. Use TextEncoder for arbitrary text.
• “Base64 makes data safe.” It only makes bytes text-safe. It is not encryption, signing, escaping, or validation.
• “A decoder always throws on bad bytes.” The default inserts �. Add { fatal: true } when invalid input should stop.
• “Hex and Base64 are interchangeable.” Both display bytes, but hex is two characters per byte while Base64 is denser and chunked.

Key takeaways

• JavaScript strings are UTF-16 code units; most storage and transport APIs work with bytes.
• TextEncoder always writes UTF-8; TextDecoder reads bytes with labels and options.
• UTF-8 uses one to four bytes per code point; byte length is not string length.
• btoa and atob use binary strings, so encode Unicode text to bytes first.
• Base64 and hex represent bytes as text. They are reversible encodings, not encryption.
• Native Uint8Array Base64 and hex helpers are useful, but feature-detect for Node 22 and older browsers.

Up next: memory management and garbage collection, where the lifetime of objects, buffers, and references becomes the main story.