Stop estimating tokens

In one sentence

A token is the unit a model actually reads, generates, and bills for — usually a few characters, never reliably a word.

Why it matters

Every limit you will hit is denominated in tokens: context windows, rate limits, output caps, and the invoice. Reasoning in characters or words means being wrong by a factor that changes per language and per input.

Three concrete consequences:

• A 100k-character document does not fit in a 100k-token window. English averages roughly 4 characters per token, so it is closer to 25k tokens — but JSON, code, and non-Latin scripts blow that ratio apart.
• Truncating input by character count can cut a multi-byte character in half and produce a token sequence the model has never seen.
• Cost forecasting from word counts is off by 30% or more, and always in the direction that surprises finance.

How tokenization works

Modern models use byte-pair encoding (BPE) or a close relative. Training starts from bytes and repeatedly merges the most frequent adjacent pair into a new token. Common words survive as single tokens; rare ones fragment.

The practical result is that frequency in the training corpus determines cost:

Two details that catch people out:

1. The leading space belongs to the token. " the" and "the" are different tokens. This is why a prompt ending in a trailing space can degrade output — you have forced the model into a token boundary it rarely saw in training.
2. Tokenizers are model-specific. A count from one provider's tokenizer is an estimate, not a fact, for another's.

Counting them

Never estimate when you are enforcing a limit. Count with the tokenizer that belongs to the model you are calling.

import Anthropic from '@anthropic-ai/sdk'

const anthropic = new Anthropic()

// Server-side count, authoritative for this model
const { input_tokens } = await anthropic.messages.countTokens({
  model: 'claude-opus-4-5',
  messages: [{ role: 'user', content: document }],
})

if (input_tokens > BUDGET) {
  // Trim on a semantic boundary, not a character offset
}

For a rough local estimate with no network call, js-tiktoken (OpenAI) or @huggingface/transformers tokenizers are close enough for capacity planning — just not for enforcement.

Budgeting a request

Input and output are usually priced differently, and output is the expensive one (often 3–5×). A working budget has four parts:

• System prompt — fixed, and a candidate for prompt caching if it is large.
• Retrieved context — the elastic part. This is where chunking and top-k decisions land.
• Conversation history — grows without bound unless you compact it.
• Reserved output — subtract max_tokens from the window before you decide how much context fits. Forgetting this is the single most common cause of "the model got cut off mid-sentence".

Common pitfalls

• Counting the prompt but not the tools. Tool and function schemas are serialized into the request and count as input. A dozen verbose tool definitions can cost more than the user's question.
• Assuming the window is usable end to end. Recall degrades well before the documented limit — see long context processing.
• Charging users per message. Message length varies by an order of magnitude. Meter tokens, or you will subsidise your heaviest users.
• Ignoring reasoning tokens. Reasoning models emit thinking tokens you pay for and usually never display.

Further reading

• OpenAI — What are tokens and how to count them.
• Anthropic — Token counting.
• Sennrich et al. — Neural Machine Translation of Rare Words with Subword Units, the BPE paper.