Mixture of experts

In a conventional model, known as dense, all the parameters take part in computing each token. A mixture of experts, often shortened to MoE, replaces some of the Transformer's layers with several “experts”, small networks placed side by side. For each token, a router picks a small number of experts judged relevant, and only those are computed. Mixtral, DeepSeek and some versions of Qwen or Llama are built on this principle.

There are therefore two figures to tell apart: the total number of parameters and the number of active parameters. A model with a high total but few active parameters generates its answers almost as fast as a small dense model, while benefiting from the knowledge of a much larger one. This is what makes very large models possible at a reasonable computing cost.

The catch is memory. Even if only a few experts are working at any given moment, all of them have to be loaded: an MoE needs as much VRAM as its total number of parameters requires. To choose a machine, look at the total to size the memory and at the active parameters to estimate the speed. MoE models make good use of a large amount of VRAM spread across several GPUs, and some tools can keep part of the experts in the processor's system RAM, at the cost of lower speed.

We help you choose between a single card and a multi-GPU setup depending on the models, dense or MoE, you want to run.