Ollama

Ollama is a free tool that makes it easy to use large language models locally. A single command is enough to download an open model, such as Llama, Mistral, Qwen, Gemma or DeepSeek, and then chat with it in a terminal. It runs on Linux, Windows and macOS and automatically detects compatible graphics cards.

Ollama relies in particular on the llama.cpp engine and on models in GGUF format, most often quantized to reduce the memory needed. It handles loading the model onto the GPU, splits the layers between VRAM and system RAM if the card is too small, and exposes a local API. Other software, such as Open WebUI, code editors or automation tools, can therefore connect to it just as they would to an online service, without the data leaving the machine.

To get the most out of it, the model must fit entirely in VRAM: as soon as part of it spills over into system RAM, speed drops sharply. On a 16 GB card, aim for models of up to about 14 billion parameters with 4-bit quantization. With two cards, Ollama can split a larger model. It is ideal for personal use or a small team; to serve many simultaneous users, vLLM is often a better fit.

Our AI workstations come with Ollama already installed and configured, together with Open WebUI, so you can get started the first time you switch on.