Token Counter
A language model does not count in words but in tokens, and the difference is larger than people expect: text in languages other than English costs more tokens per word, and code, JSON and emoji behave differently again. This tool counts locally using the same OpenAI encodings — cl100k_base for GPT-3.5 and GPT-4, and o200k_base for GPT-4o and newer — rather than the common rule of thumb of four characters per token. You see not only the number but also how your text breaks into individual token pieces. You can also load a text file such as TXT, Markdown, JSON or CSV; the file never leaves your browser. For Anthropic and Google the result is an indication, because they use their own tokenizers. Use is free and unlimited, including commercially — no account, no licence, no cap on how often.
What people use this for
- Checking whether a system prompt fits the context window
- Comparing two phrasings on token cost
- Estimating up front what a batch of a thousand documents will cost
What others think of this tool
No reviews for this tool yet. Yours would be the first.
Do you work with Westcube?
Leave a review on Google tooFrequently asked questions
Is this the exact token count for my AI model?
For OpenAI models using the selected encoding: yes, exactly. This is the same tokenizer library OpenAI publishes itself, not an approximation. Choose cl100k_base for GPT-3.5 and GPT-4, and o200k_base for GPT-4o and newer models. For other providers it is an indication: Anthropic and Google each use their own tokenizer, which arrives at a different number for the same text. In practice, for ordinary prose they land within roughly ten percent of each other, which is ample for a cost estimate, but do not settle accounts to the token with it.
Roughly how many tokens is a word?
The rule of thumb you read everywhere — about four characters per token, or three quarters of a word per token — holds reasonably for English prose and much less well for anything else. Languages that build long compound words are more expensive, because a word like the Dutch "verzekeringsmaatschappij" is chopped into several pieces where English uses a single token. Code, JSON and URLs also fare badly because of all the punctuation. Emoji and non-Latin scripts can cost several tokens per character. That is precisely why counting beats estimating, especially with a tight context window or a large volume.
Is my text sent to an AI provider?
No. The text is processed entirely in your browser and is not sent to our server, OpenAI or any other AI provider. Text from a locally loaded file stays local too.
Is there a maximum text length?
Yes, fifty thousand characters at a time. That is generous: roughly a document of fifteen to twenty pages, and more than enough for virtually any prompt. The limit exists to stop a single request occupying the server for minutes. For a larger document, split it and add the counts — tokenizing is close to additive, with at most a few tokens of difference at the split points. There is no daily cap and no paid tier with a higher limit.
Do input and output tokens both count towards the cost?
Yes, and that is often forgotten. Providers bill input and output separately, with output usually considerably more expensive per token — a factor of four or five with some models. This tool counts what you put in. For a full cost estimate, add the expected response, and for a multi-turn conversation also the entire history that is resent each time. That last one is the most underestimated cost in a chat application.