F.1: Tokens and the Architecture of Language

What Are Tokens?

Large language models do not process language the way humans do. They cannot read words directly — instead, they break text down into tokens, which are the basic computational building blocks of all language processing.

A token can represent:

• A single character (like x or ,)
• A word fragment (like un or tion)
• A complete word (like hello)
• A short phrase or punctuation mark

The process of breaking text into tokens is called tokenization, and it happens before anything else. When you send a message to an AI model, your raw text is first converted into a sequence of numeric IDs — one ID per token — and those numbers are what the model actually processes.

Why Not Just Use Words?

You might wonder why models do not simply split text on spaces and use whole words as their unit of processing. There are several reasons:

1. Vocabulary size. English alone has over 170,000 words in current use. Add technical jargon, proper nouns, misspellings, and slang, and the vocabulary becomes unmanageable. Tokens let models work with a smaller, fixed vocabulary (typically 30,000-100,000 tokens) that can still represent any text.

2. Morphology. Words like "running," "runner," and "runs" share a root. Subword tokenization can represent these as combinations of shared pieces, helping the model recognize the relationship.

3. Unknown words. A word-level model would fail on any word it never saw during training. A subword tokenizer can break unfamiliar words into known fragments, maintaining at least partial understanding.

The Numbers Behind Tokenization

Understanding the relationship between text and tokens is essential for practical AI work, because most AI services charge per token and context windows are measured in tokens.

These are averages for English text. Common words like "the" or "and" are often single tokens. Longer or less common words get split into multiple tokens. For example, "tokenization" itself is typically broken into 3 tokens: token, ization (or similar fragments depending on the model).

Cross-Language Tokenization

One of the most important practical implications of tokenization is that it is not language-neutral. Models trained primarily on English text develop tokenizers that are highly efficient for English but significantly less efficient for other languages.

Consider this: a sentence in English might require 15 tokens, but the same meaning expressed in Telugu, Hindi, or Chinese could require 30-50 tokens — even though the translated text might have fewer characters. This happens because:

• The tokenizer was trained primarily on English text, so English words and subwords are heavily represented in the vocabulary
• Characters from other scripts are less efficiently encoded
• Some languages require multiple tokens per character

This disparity has real consequences:

• Cost: API usage is billed per token, so non-English users pay significantly more for the same work
• Context window usage: Less efficient tokenization means you can fit less content from non-English languages into the same context window
• Performance: Models may produce lower-quality outputs in languages where tokenization is less efficient, because less semantic information is packed into each token

How Tokenization Connects to Model Architecture

The tokens produced by the tokenizer do not go directly into the model's reasoning layers. First, each token ID is converted into an embedding — a high-dimensional vector (a list of numbers) that represents the token's meaning. The model learned these embeddings during training, and similar tokens have similar embeddings.

The flow looks like this:

1. Raw text → Tokenizer → Token IDs (a sequence of integers)
2. Token IDs → Embedding layer → Vectors (one per token)
3. Vectors → Transformer layers → Predictions (probability of the next token)

This is why tokenization is the "architecture of language" in the context of AI — it is the bridge between human-readable text and the mathematical representations that the model actually computes with.