vLLM

vLLM is free software, born at the University of California, Berkeley, that specialises in inference for large language models. Where Ollama aims for simplicity on an individual workstation, vLLM aims for throughput: it is used to run corporate assistants, internal APIs or services that receive many simultaneous requests.

Its flagship technique is called PagedAttention. It stores the cache linked to the context window in small blocks, much like an operating system's virtual memory, which avoids wasting VRAM. Combined with continuous batching, which groups requests from several users into the same computations, it lets you serve far more conversations in parallel with the same GPU. vLLM can also split a model across several cards and offers an API compatible with most tools on the market.

vLLM is mainly used on Linux with recent NVIDIA cards. By default, it reserves most of the VRAM for its cache: the available memory directly determines the number of simultaneous users and the possible context length. For a model split across several GPUs, a fast link between the cards, such as NVLink on models that have it, improves performance. Expect a little more configuration than with Ollama.

For a team or a business, we can size a multi-GPU setup suited to vLLM based on the expected number of users.