Memory bandwidth
The amount of data a GPU's memory can exchange with the chip each second, in GB/s or TB/s. A key factor in AI speed.
Memory bandwidth measures the amount of data a graphics card's memory can transfer to the graphics processor in one second. It is expressed in gigabytes per second (GB/s) or, on data center GPUs, in terabytes per second (TB/s). It depends on the memory type, its frequency and the width of the bus connecting it to the GPU.
A GPU can have enormous computing power, yet it stalls if it is waiting for its data. This is often the case in artificial intelligence: during text generation, each new token forces the GPU to reread almost all of the model's parameters. Generation speed therefore depends more on bandwidth than on raw computing power. Put simply, dividing the bandwidth by the size of the model in memory gives an upper limit on the number of tokens per second for a single request.
On a product page, bandwidth is often listed next to the VRAM capacity. To compare two cards intended for AI, look at both together: capacity determines which models can be loaded, bandwidth determines how responsive they are. HBM memory offers the highest bandwidth, ahead of GDDR7 and GDDR6X. For mining, some algorithms are also memory-bound, which explains the interest in memory clock tuning.
Our NVIDIA H200 page shows what very high bandwidth means for AI workloads.