Model parameters
Also called: billion parameters, model weights, parameter count
Numbers adjusted during training that hold an AI model's knowledge. Their count, in billions, shows the model's size and the memory it needs.
An AI model's parameters are the numbers, also called weights, that connect its artificial neurons to each other. At the start they are random; during training they are adjusted a huge number of times until the model produces good answers. Once training is finished, they contain everything the model has learned. Their count is used to describe the model's size: 8B, 70B or 405B, where B stands for billion.
The more parameters a model has, the more knowledge it can retain and the more finely it can reason, but the more memory and computing power it needs. Each parameter takes up 2 bytes at 16 bits: a 70-billion-parameter model therefore weighs about 140 GB, and around 40 GB once quantized to 4 bits. Mixture-of-experts models distinguish between the total number of parameters and the number of active parameters for each token, which makes them faster than their size suggests.
To choose a machine, start from the model you are aiming for. At 4 bits, count a little more than half the number of billions of parameters in gigabytes, plus a margin for the context. A 16 GB card runs models of up to about 14 billion parameters, 32 GB lets you go up to around 30 billion, and 70-billion-parameter models need at least 48 GB. A recent, smaller model can actually beat a bigger but older one.
Our AI workstations range from a single 16 GB card to several 96 GB professional cards: tell us which models you are aiming for, and we will point you to the right configuration.