Transformer
Also called: attention mechanism, Transformer architecture
The Transformer is the neural network architecture behind LLMs: its attention mechanism links each word in a text to the others to grasp its meaning.
The Transformer is a neural network architecture introduced in 2017 by Google researchers in the paper “Attention Is All You Need”. Today it underpins almost all large language models, such as GPT, Llama, Mistral or Gemini, and is also used for translation, computer vision and speech recognition.
Its central idea is the attention mechanism. The text is split into tokens, each token is turned into a vector, then the model calculates, for each one, the importance of all the others. In “the miner is getting hot because it is running at full load”, attention helps the model understand that “it” refers to the miner. These calculations are done in parallel, as large matrix multiplications, which explains why GPUs and their Tensor Cores are so efficient at running a Transformer. A model stacks dozens of layers of this kind.
Two characteristics of a model matter when choosing a machine. Its number of parameters sets the memory needed to load it. Its context window also weighs in, because the cost of attention and the cache memory grow with the length of the text being processed. A GPU with plenty of VRAM and high memory bandwidth will produce more tokens per second.
Our page on how an LLM works shows step by step what happens inside a Transformer, from splitting text into tokens to choosing the next word.