Token

A token is the small piece of text a language model works with. Before it is processed, any text is split up by a tool called a tokenizer: a common word often forms a single token, while a long, rare or compound word is cut into several pieces. Punctuation, spaces and numbers have their own tokens too. Each model has its own vocabulary, ranging from a few tens of thousands to more than a hundred thousand tokens.

Everything is counted in tokens. The model generates its answer one token at a time, each one depending on all those before it. The context window, meaning the amount of text the model can take into account at once, is expressed in tokens. And online AI services generally charge by usage, according to the number of tokens sent and received. In English, a text usually contains somewhat more tokens than words.

To judge a local AI machine, the key indicator is generation speed in tokens per second. It depends mainly on the GPU's memory bandwidth and the size of the model. Above twenty or so tokens per second, text appears faster than you can read it. Also look at prompt processing speed, which matters when you submit long documents for analysis.

With an AI workstation at your place, there is no more per-token billing: you use your model as much as you like, and the running cost comes down to the electricity consumed.