Token Counter: Estimate Tokens Before You Send a Prompt

There are two moments when you need to know a token count, and both of them are before you press send. The first is when a prompt might not fit, because the model will not politely warn you, it will truncate silently or fail outright. The second is when the prompt is about to run a million times, and 200 wasted tokens becomes a permanent tax on every call.

This article covers how to estimate a count in your head, when that estimate will betray you, and how to budget a context window properly. Our Token Counter gives you the exact figure for any text, under the tokenizer of your choice.

TL;DR

Estimate with characters, not words: divide by about 5.4 for English prose, but only 2.8 for code or JSON.
The old "4 characters per token" rule is wrong in both directions, depending on what you are sending.
Tokenizers disagree. The same Indonesian paragraph is 72 tokens under o200k but 92 under cl100k, a 28% penalty, while English is identical under both.
Non-English text is systematically more expensive to express, because the tokenizer's merges were learned mostly on English.
Your context window is a budget: every token of prompt is one the model cannot use to reply.

Estimating in your head

Characters are a better basis than words, because a token is a chunk of characters and has no idea where your word boundaries are. The estimate is:

T \approx \frac{characters}{k}

where $k$ is the characters-per-token ratio for the kind of text you are sending. Measured with the o200k tokenizer that GPT-4o and 4.1 use:

Content	Characters per token $k$	Tokens per word	Divide characters by
English prose	5.4	1.1	5
Indonesian prose	4.7	1.5	4.5
Source code	2.8	2.7	3
JSON payload	2.7	n/a	3

Those figures are measured on real samples, not folklore, and the spread is the point: the same number of characters of JSON costs roughly twice as many tokens as English prose. Structure is what you are paying for. Braces, quotes, indentation and closing tags are all tokens, and none of them mean anything to your user.

Because an estimate that is too low is the dangerous direction, give it a margin before you rely on it:

T_{safe} = ⌈ 1.15 \cdot \frac{characters}{k} ⌉

Tokenizers disagree, and not evenly

There is no universal token. Each model family ships its own vocabulary, learned from its own corpus, so the "length" of your prompt is a property of where you send it. Here is the same set of texts measured under the two OpenAI tokenizers:

Identical texts, two tokenizers. English, code and JSON are all but unchanged. Indonesian prose costs 92 tokens under cl100k against 72 under o200k, because the newer vocabulary learned merges that cover Indonesian word shapes.

That Indonesian result is worth dwelling on, because it generalizes. A tokenizer's merges are learned from a corpus that is overwhelmingly English, so English gets compressed efficiently and everything else pays a surcharge in tokens for saying exactly the same thing. Moving from cl100k to o200k narrowed that gap considerably, which is a quiet but real cost reduction for anyone working in a language other than English. If your users write in Bahasa Indonesia, the model you pick changes your bill for reasons that have nothing to do with the model's price per token.

Which is the practical argument for measuring rather than assuming. Our Token Counter supports the GPT, Claude and Llama tokenizers and highlights each token in place, so you can see exactly where a long prompt is being spent.

The context window is a budget

A context window is not a capacity you fill, it is a budget you divide. Input and output share it:

T_{system} + T_{history} + T_{docs} + T_{out} \leq C

so the room the model has to answer in is simply whatever you did not spend on the prompt:

T_{out}^{m a x} = C - (T_{system} + T_{history} + T_{docs})

A 128K window, spent. Retrieved documents are usually the largest and least disciplined item, and every token they take is a token the model cannot use to answer.

This is where silent truncation comes from. Stuff enough retrieved context into the prompt and $T_{out}^{m a x}$ goes to zero, at which point the model has nowhere to put its answer. Worse, some stacks quietly drop the middle of an over-long prompt, so the model does not fail, it just answers from material you did not realize was missing.

Budgeting retrieval

If you are doing RAG, the document term is under your control and worth computing rather than guessing. Splitting a document of $D$ tokens into chunks of size $c$ with an overlap of $o$ gives:

n_{chunks} = ⌈ \frac{D - o}{c - o} ⌉

and if you retrieve the top $k$ of them, the prompt carries $k \cdot c$ tokens of context. That is the whole retrieval budget, and it is why the overlap parameter matters more than people expect: an overlap approaching the chunk size makes the denominator small and the chunk count explode, so you pay for the same sentences many times over in your embedding bill.

Retrieved context usually starts life as PDFs, scans or web pages. Get it into clean text first, with Free OCR or Web to Markdown, and you will typically find the token count drops by more than half without losing a word of meaning, because most of what was there was markup.

If you host the model yourself

On your own hardware, tokens stop being an invoice and become a memory constraint. Every token in the context occupies KV cache on the GPU for as long as the request is alive, and that cache is usually the thing that runs out first. The arithmetic is in how much VRAM you need to run an LLM, and the LLM VRAM Calculator turns your context length and concurrency into a GPU size.

Either way, the discipline is identical. Count first, then send. Netra helps teams fine tune, accelerate and deploy models on their own infrastructure, where a prompt you trimmed by 30% is 30% more traffic the same GPU can serve.

FAQ

Can I just divide by four? You will underestimate English by about 25% and overestimate code by about 30%. Use 5 for prose, 3 for anything structured, and count exactly when it matters.

Does the same text cost the same on every model? No. Different tokenizers give different counts for identical text, and the gap is much larger for non-English text than for English.

Why is my code prompt so expensive? Because indentation, brackets and punctuation are all tokens. Code is roughly twice as token-dense as prose, character for character.

What happens if I exceed the window? Depending on the stack, an error or a silent truncation. The silent case is the dangerous one, since the model will answer confidently from a prompt you did not fully send.

Free tools from Netra

All of these run in your browser. No signup, and your text is not uploaded to us.

Token Counter: the exact count, per tokenizer, with each token highlighted so you can see where they go.
Web to Markdown: strip a page to clean Markdown before you paste it into a prompt.
Free OCR: extract text from scans and PDFs so it can be counted at all.
LLM VRAM Calculator: on your own GPU, tokens become KV cache. This sizes it.
Fine-Tuning Memory Calculator: memory for training rather than serving.
Limbus: local-first image segmentation.
sam3.c: SAM3 inference in pure C.

References

tiktoken, the reference BPE tokenizer for the GPT family (cl100k and o200k).
Sennrich et al., Neural Machine Translation of Rare Words with Subword Units.
Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages, on why non-English text costs more tokens.

Blog / Guides