If you’ve used an AI chatbot’s pricing page or API documentation, you’ve probably run into the word “token” without much explanation of what it actually means. Tokens are the basic unit AI language models use to read and generate text — and understanding them helps explain everything from AI pricing to why a model sometimes “runs out of room” in a long conversation.
What Is a Token?
A token is a chunk of text — which could be a whole word, part of a word, a single character, or even punctuation — that a language model treats as one unit when reading or generating language. Before a model can process any text, it has to break that text down into tokens through a process called tokenization.
As a rough rule of thumb, one token is roughly four characters of English text, or about three-quarters of a word. So a 1,000-word article might translate to around 1,300 tokens once tokenized. Different languages tokenize differently — languages with more complex scripts or less common vocabulary in the model’s training data often require more tokens to represent the same amount of meaning.
Why Don’t Models Just Process Whole Words?
Breaking text into sub-word tokens rather than whole words solves a practical problem: no model could realistically have a dictionary entry for every possible word, especially rare words, names, typos, or made-up terms. By breaking words into smaller common pieces, a model can represent virtually any text — including words it has never seen before — by combining familiar sub-word chunks. For example, the uncommon word “tokenization” might be split into pieces like “token,” “iz,” and “ation.”
Why Tokens Matter for Using AI Tools
Tokens show up in a few practical ways once you’re actually using AI tools:
- Pricing: Most AI APIs charge per token processed (both the tokens you send in and the tokens the model generates back), which is why longer prompts and longer responses cost more.
- Context window limits: Every model has a maximum number of tokens it can consider at once — called its context window. Once a conversation or document exceeds that limit, older content gets dropped or the model can’t process it at all.
- Response speed: Models generate text one token at a time, so longer responses take measurably longer to produce.
How Tokenization Affects What AI Gets Right (and Wrong)
Tokenization also explains some famously odd AI behavior — like a model struggling to count the letters in a word, or mishandling certain math problems. Because the model sees chunks of characters rather than individual letters, tasks that require letter-by-letter precision can trip it up in ways that seem surprising given how fluent its writing otherwise is.
The Bottom Line
Tokens are the basic currency of how AI language models read, think about, and generate text. Understanding roughly how many tokens your text takes up — and that both pricing and context limits are measured in tokens, not words — makes AI tool costs and limitations much less mysterious.