Core Concepts

What Are Tokens in AI?

Tokens are the units of text that a language model processes — often words or fragments of words. Text is broken into tokens before the model reads it, and the model generates its output one token at a time. Token counts determine how much text fits in a model’s context window and are used to measure and price usage.