Learn / What is a token?
What is a token? How LLM tokenization works
A token is the basic unit of text an LLM processes — usually a sub-word chunk, roughly three-quarters of a word in English. Models convert text to tokens with a tokenizer, most commonly one based on byte-pair encoding (BPE). Because each model family uses its own tokenizer, the same text can be a different number of tokens in GPT, Claude, or Llama — which matters because you pay per token and context limits are measured in tokens.
How tokenization works
A tokenizer maps text to integers from a fixed vocabulary. Most modern tokenizers use byte-pair encoding, introduced for neural translation by Sennrich, Haddow, and Birch (2016): starting from characters, the most frequent adjacent pairs are merged repeatedly into larger sub-word units. Common words end up as a single token, while rare or novel words are split into several pieces — which is how a fixed vocabulary can still represent any text.
Why counts differ across models
There is no single universal tokenizer. GPT models use OpenAI's tiktoken encodings (cl100k_base for GPT-4 and GPT-3.5, o200k_base for GPT-4o and the o-series). Open models like Llama, Gemma, Qwen, and Mistral each ship their own tokenizer, and Claude and Gemini use proprietary ones. The same sentence can therefore be, say, 18 tokens for one model and 21 for another.
| Model family | Tokenizer | Counting |
|---|---|---|
| GPT-4o, o-series | tiktoken o200k_base | Exact, offline |
| GPT-4, GPT-3.5 | tiktoken cl100k_base | Exact, offline |
| Llama, Gemma, Qwen, Mistral | Model-specific (BPE / SentencePiece) | Exact, from the model's tokenizer |
| Claude, Gemini | Proprietary | Estimate offline; exact via the provider API |
Why it matters
Tokens are the currency of LLMs. You are billed per input and output token, so token count drives cost. A model's context window — how much it can read at once — is measured in tokens. And more tokens mean more compute and a longer stream, so counting tokens up front helps you predict cost, fit within limits, and estimate latency.
Try it yourself
Free tool
Token counter
Paste any text and see the exact token count for GPT, Llama, Gemma, Qwen, and Mistral, plus an estimate for Claude.
Frequently asked questions
How many tokens is a word?
As a rough guide, one token is about three-quarters of a word in English, so 100 tokens is roughly 75 words. The exact ratio depends on the tokenizer and the text, and it differs for code, other languages, and unusual words.
Why do GPT and Claude count tokens differently?
Each model family uses its own tokenizer with its own vocabulary, so the same text splits into different tokens. GPT uses OpenAI's tiktoken encodings, while Claude, Gemini, and open models each use their own, which is why counts do not match exactly.
Why does token count matter?
Tokens are the unit of both pricing and context limits: you are usually billed per token, and a model's context window is measured in tokens. More tokens also mean more compute per request, so token count affects cost, capacity, and latency.
References
- Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. arXiv:1508.07909.