AI language models don't see words. They split text into tokens, small pieces that can be a whole word, part of a word, a space or a punctuation mark. Context windows, rate limits and API prices are all measured in tokens, so it helps to know roughly how many you are using.
The rule of thumb
For typical English prose, 1,000 tokens is about 750 words, or 1 token is roughly 0.75 words. Another common approximation is about 4 characters per token.
750 words to tokens
750 words ÷ 0.75equals1,000 tokens
200,000 tokens to words
200,000 tokens × 0.75equals150,000 words
That second line is a useful sense of scale: a 200,000-token context window holds roughly 150,000 words of English text, around the length of a long novel.
Why words and tokens differ
Common short words like "the" or "and" are usually one token each. Longer or rarer words are split into several pieces. Numbers, code, URLs and unusual names can use a lot of tokens for their length.
When the rule of thumb breaks down
- Code: brackets, indentation and variable names often use more tokens per word than prose.
- Other languages: many languages use more tokens per word than English, sometimes considerably more.
- Data formats: JSON, tables and lists of numbers can be token-heavy.
- Different models: each provider uses its own tokenizer, so the same text can produce different counts.
When you need an exact count
For planning and rough cost estimates, the 0.75 rule is normally close enough. When you are near a context limit, or billing a client for usage, use the provider's own token counting tool or library for the exact number.
What it means for cost
Tokens are what you pay for, so the conversion matters when budgeting. A 10,000-word English document is roughly 13,300 tokens. Sending it as input to a model priced at $2 per million input tokens costs about $0.027. The same document translated into a language that tokenizes less efficiently could cost two or three times as much, for exactly the same content.
Output usually costs several times more per token than input, so a long answer can cost more than a long prompt. Asking for concise answers, or for structured data with short field names, is one of the simplest ways to cut spending.
Different models, different counts
Each model family uses its own tokenizer, so the same text does not produce the same number of tokens everywhere. Even within one provider, a new model generation can change it. When Anthropic released Claude Sonnet 5, independent testers reported that its tokenizer produced roughly 30% more tokens for the same text than the previous generation. A price per token is only comparable across models if you also compare how many tokens each needs for your content.
Things that quietly add tokens
- JSON and other structured formats: quotes, braces and repeated field names all count.
- Long system prompts and tool definitions that are resent with every request.
- Conversation history: in a chat, every earlier message is usually sent again with each new one.
- Whitespace, indentation and formatting in code.
- Hidden reasoning in models that think before answering, which is billed as output.
Quick reference (English prose, about 0.75 words per token)
- 1,000 tokens: about 750 words, roughly one and a half pages.
- 10,000 tokens: about 7,500 words, a long report.
- 100,000 tokens: about 75,000 words, the length of a novel.
- 1,000,000 tokens: about 750,000 words.
A worked budget
Suppose you want to summarise 1,000 documents of about 3,000 words each, with a 225-word summary for each. At about 0.75 words per token, each document is roughly 4,000 input tokens and each summary 300 output tokens: 4 million input and 0.3 million output tokens in total. On GPT-6 Luna ($0.10 / $0.50 per million) that costs about $0.55. On Claude Sonnet 5 or GPT-6 Sol ($2 / $10) it costs about $11. Either way the job is cheap, but at a million documents the same choice becomes $550 against $11,000.
Before running a large job, test a sample of 20 documents on each model and compare quality. Use the API's reported token counts from that sample, not the rule of thumb, to set the final budget.
Tip: Remember that both directions count. Your prompt, any documents you include and the model's reply all use tokens, and replies are usually priced higher.
Questions people ask
- Why do some languages use more tokens?
- Tokenizers are built mostly from English-heavy text, so common English words are often a single token, while words in many other languages are split into several pieces. The same meaning can therefore cost noticeably more tokens, and more money, in some languages.
- How do I get an exact token count?
- Use the provider's own tokenizer or token-counting tool for the model you will use, or read the token counts returned in the API response. The rule of thumb is only for early estimates.
- How many words is 1,000 tokens?
- About 750 words of typical English prose. Another rule of thumb is about 4 characters per token.
- How many words fit in a 200,000-token context window?
- Roughly 150,000 words of English, around the length of a long novel.
- Why do code and other languages use more tokens?
- Tokenizers split rarer words, symbols and non-English text into more pieces, so the same number of words can need many more tokens.
- Do all AI models count tokens the same way?
- No. Each model family has its own tokenizer, and new generations can change it. Compare token counts for your own text, not just the price per token.