Quantization
Also called: quantisation, 4-bit, 8-bit
A technique that stores an AI model's weights in fewer bits, 8 or 4 instead of 16, to cut the memory needed with little loss of quality.
Quantization means converting an AI model's parameters into a more compact format. A model is usually published at 16 bits per parameter; once quantized, it drops to 8 bits, 4 bits, sometimes less. The model stays the same, but each number is rounded to take up less space.
The gain in memory is immediate. An 8-billion-parameter model takes up about 16 GB at 16 bits, 8 GB at 8 bits and around 5 GB at 4 bits. Since text generation is limited mainly by how fast the GPU reads these weights, a quantized model also responds faster. The trade-off is a slight drop in quality, barely noticeable at 8 bits, generally moderate at 4 bits, and more pronounced below that. Several methods exist, and ready-to-use models are often offered in 4-bit by default.
In practice, quantization decides what fits on your graphics card. With 16 GB of video memory, a 14-billion-parameter model at 4 bits fits comfortably; with 32 GB, you can aim for models of around 30 billion parameters. Always keep a margin for the context window, which also takes up memory. A bigger but quantized model often gives better results than a small model at full precision.
Our AI workstations come with Ollama and Open WebUI installed: you can download quantized models and use them straight away.