Episode 6: What is a token?

1. 1 A token is a chunk of textIt can be a whole word, part of a word, a single character, or even a byte.Sources: OpenAI and Anthropic docs.
2. 2 One word can be several tokensOpenAI's docs show the string " tokenization" splitting into " token" and "ization". Smaller pieces help models handle uncommon words.Other models may cut the same word differently.
3. 3 Your prompt and the answer share one limitThe context window covers everything the model can reference, including its own response. Both are counted in tokens.Try: Paste only the pages that matter, not the whole file. (General good practice: Anthropic notes that more context is not automatically better.)
4. 4 Different models, different countsHow many characters a token covers depends on the model and the language. Never assume one model's count applies to another.
5. 5 A rough guide, not a ruleFor English, OpenAI's docs say 1 token is roughly 4 characters or 0.75 words. Anthropic's glossary says about 3.5 English characters for earlier Claude models.Estimates only. For a closer count, use the provider's token counter for the model you will use.

Tap a step when you have done it. Your ticks stay on this device.