Inference
The stage where an already trained AI model is used to respond: generating text, analysing a document or an image. It is the everyday use of AI.
Inference refers to using an AI model once its training is complete. Every time you ask an assistant a question, have it summarise a document or describe an image, the model performs inference: it applies what it has learned, without changing its parameters. It is the opposite of training, which is used to build the model and requires far heavier resources.
For a language model, inference happens in two stages. First, the model reads your whole request in one go: this stage relies mainly on computing power. Then it generates the answer token by token, rereading all its parameters in memory each time: this stage depends mainly on the bandwidth of the GPU's memory. That is why a GPU with very fast memory responds more quickly, even with comparable computing power.
The good news: inference is within reach of a single machine. For personal use or a small team, a workstation with one or two graphics cards is enough for many models. To serve dozens of simultaneous users, you move up to a multi-GPU server and optimised software such as vLLM. The criteria to check are video memory, its bandwidth, the supported precisions and power consumption.
Our AI workstations and GPU servers are designed for local inference: your data stays on your premises, with no subscription to an online service.