Tokenization & Embeddings

Why Tokenization Matters

Tokenization Pipeline

LLMs don't process text — they process tokens. Tokenization is the bridge between human language and the mathematical operations inside a neural network. Your choice of tokenizer directly impacts model quality, cost, and context window efficiency.

Common Tokenization Algorithms

• BPE (Byte Pair Encoding): Used by GPT models. Iteratively merges the most frequent pair of bytes/characters into a single token. Good balance of vocabulary size and coverage.
• WordPiece: Used by BERT. Similar to BPE but uses likelihood instead of frequency for merging decisions.
• SentencePiece: Language-agnostic tokenizer that works directly on raw text (no pre-tokenization). Used by LLaMA and T5.
• tiktoken: OpenAI's fast BPE implementation. The standard for GPT-3.5/GPT-4 token counting.

Token Embeddings

After tokenization, each token ID is mapped to a dense vector (embedding) through a lookup table. These embeddings are the actual inputs to the Transformer. They encode semantic meaning — similar words have similar vectors.

Why This Matters for Cost

OpenAI and other providers charge per token. A poorly tokenized prompt wastes money. For example, the word "tokenization" might be 1 token or 3 tokens depending on the tokenizer.