Token

A token is a discrete unit from a tokenizer vocabulary, represented as an integer ID that an LLM processes and predicts.

Details

In LLMs, objectives like next-token prediction during training operate over sequences of tokens; during inference, generated tokens are decoded back into text by the tokenizer. Token counts are the unit for both context size limits and inference cost billing, and are tokenizer/model-dependent.

Note: In other contexts, "token" can mean an authentication credential; here it refers to the model's text units.

Examples

• The text "unbelievable" might be one token or multiple subword tokens depending on the tokenizer.
• Special tokens can mark boundaries (begin/end of sequence) or structure (message separators).