Encoding for developers: UTF-8, Base64, and URL encoding

Published 9 June 2026 · 8 min read

Encoding bugs have a signature. Café renders as Café. A Base64 decode fails on invalid input. A URL containing another URL breaks in a way that only shows up in production.

All three come from the same root confusion: encoding is not encryption, not compression, and not one thing. UTF-8, Base64, and percent-encoding solve three unrelated problems, and using one where another is needed produces exactly these symptoms.

Characters versus bytes

A computer stores bytes. Text is characters. A character encoding is the mapping between them, and every text bug in this article is a place where two parties disagreed about which mapping is in use.

Unicode assigns every character a number — a code point. U+0041 is A, U+00E9 is é, U+1F600 is a grinning face. Unicode says nothing about how to store those numbers as bytes. That is what UTF-8 does.

UTF-8

UTF-8 is a variable-length encoding: one byte for ASCII, two to four for everything else. Its design is the reason it won.

• ASCII is unchanged — any ASCII file is already valid UTF-8, which made adoption free.
• It is self-synchronising. Continuation bytes are distinguishable from lead bytes, so a decoder can recover after a corrupt byte instead of garbling the remainder.
• It is endianness-free, unlike UTF-16, so no byte order mark is needed to interpret it.

The consequence developers hit most: a character is not a byte. "café" is five characters and six bytes. Truncating a UTF-8 string at a byte boundary can split a character in half and produce a replacement character or invalid output. Any length limit measured in bytes needs to respect character boundaries.

Mojibake

The classic symptom: text encoded as UTF-8 and decoded as Latin-1. The é is two bytes, 0xC3 0xA9; interpreted one byte per character in Latin-1 they render as à and ©.

café   → Café      (UTF-8 read as Latin-1)
naïve  → naïve
"      → “        (smart quote, three bytes)

Seeing à or †in output tells you exactly what happened. The fix is at the boundary where the wrong decoding occurred — a database connection charset, an HTTP Content-Type header, a file read without an explicit encoding — not a search-and-replace on the text, which only pushes the problem downstream.

Double encoding produces the same characters run through the process twice: Café. If your mojibake looks unusually long, it has been through two passes and both need fixing.

Base64

Base64 solves a different problem: moving binary data through a channel that only reliably handles text. Email bodies, JSON string fields, data URIs, and HTTP headers all need this.

It takes three bytes at a time, splits them into four six-bit groups, and maps each group to one of 64 characters. Three bytes become four characters, so the output is always about 33% larger than the input. When the input is not a multiple of three, = padding fills the gap.

Two things follow that people repeatedly get wrong. Base64 provides no confidentiality whatsoever — a Base64 blob in a config file is not a protected secret, it is a secret in a thin disguise. And it makes data bigger, not smaller; it is not compression.

Base64URL

Standard Base64 uses + and /, both of which have meaning in a URL. Base64URL substitutes - and _ and usually drops the padding. This is what JWTs use, which is why a JWT segment pasted into a standard Base64 decoder sometimes fails.

standard:  a+b/c==
url-safe:  a-b_c

Percent-encoding (URL encoding)

A third problem again: URLs reserve certain characters as structure. The ? starts a query string, & separates parameters, / separates path segments, # starts a fragment. A value that contains one of those characters has to be escaped or it changes the meaning of the URL.

Percent-encoding replaces a character with % followed by its byte value in hexadecimal. A space becomes %20 — or, in a query string only, sometimes +, a legacy of HTML form encoding that is a genuine inconsistency in the standards.

Encoding the whole URL versus encoding one value

This is the distinction behind most URL bugs. JavaScript has two functions and they are not interchangeable.

encodeURI("https://x.com/a b?q=1&r=2")
  → "https://x.com/a%20b?q=1&r=2"      structure preserved

encodeURIComponent("https://x.com/a?q=1")
  → "https%3A%2F%2Fx.com%2Fa%3Fq%3D1"  safe as a value

Use encodeURI on a whole URL you are cleaning up. Use encodeURIComponent on anything going into a query parameter — especially another URL, which is the case where getting it wrong is most common and most damaging. A redirect_uri or callback URL passed unencoded loses everything after its first &, because the outer URL claims those parameters as its own.

Double encoding

Encoding an already-encoded string turns each % into %25. The URL still works structurally, and the value that arrives is wrong: a search for "a b" comes through as "a%20b" literally.

Seeing %25 in a URL almost always means something was encoded twice — commonly a framework that encodes automatically, plus application code that encodes as well.

Which one do I need?

• Text has to survive being stored and read back — UTF-8, and make sure both ends agree.
• Binary data has to fit inside a text field, a JSON string, or a header — Base64.
• A value has to go into a URL — percent-encoding, per component, once.
• Data has to be kept confidential — none of these. That is encryption, and it is a different subject entirely.

That last line is worth repeating because it is the most consequential confusion of the four. Base64 is not a security measure. Anything you Base64 to hide it is visible to anyone who thinks to decode it, which takes one paste into a decoder.