The HGX H200 server: eight GPUs in one machine
An 8U HGX H200 server with its cover removed, laid out like NVIDIA's reference design (the DGX H200): at the top, the GPU tray with its eight H200s, its four NVSwitches (glowing dots: NVLink traffic) and its backplane; below it, the motherboard tray, pulled out at the rear to show it; at the very bottom, the power supplies; at the front, the fans and the SSDs. GPU 1 slides out of its slot: move the slider to put it back in place, then click “Module” to open it.
A data center GPU is not a graphics card
The H200 SXM has no fan, no video output and no standard PCIe connector: it is a “mezzanine” module that screws onto an 8-GPU HGX board and is cooled by the server. Select a component to see what it does.
The GH100 chip seen from above
814 mm² of silicon: 8 compute blocks (GPCs) around the L2 cache, the HBM memory controllers on the sides, NVLink at the top, PCIe at the bottom. Click an area.
One multiprocessor (SM) out of 132
Four identical blocks, each with its own CUDA cores, Tensor Core and registers, around 256 KB of shared memory. Click a block.
A Tensor Core multiplies matrices
D = A × B + C in a single instruction: each row of A meets each column of B, and the result accumulates in D. This is the basic operation of a neural network.
The GPU we are pulling out
NVIDIA H200 SXM5 · 141 GB · 700 W
One of the eight H200 modules, with its gold-capped heatsink. It does not plug into a PCIe slot and has no cables: it sits flat on the HGX board, on two mezzanine connectors that supply it with power and data, and its heatsink presses it into place. The slider pulls it out of its slot; click “Module” to open it layer by layer.
The seven other GPUs
8 × H200 · 1.1 TB of HBM3e
Eight identical modules in two rows of four, aligned with the airflow: each one sits under a large finned heatsink through which the air from the fans flows from front to back. Together: 1.1 TB of HBM3e memory and about 32 PFLOPS in FP8 (with sparsity). The software sees eight GPUs that can share a single model.
HGX H200 board and midplane
The eight-GPU board
A large board designed by NVIDIA and supplied to server manufacturers (Supermicro, Dell, HPE, Gigabyte…): it carries the eight SXM5 slots, the four NVSwitches and the power delivery for the GPUs. At the back of the tray, a midplane connects it to the rest of the server: power, PCIe 5.0 links to the CPUs, sensors and signals. Same form factor as the HGX H100 board: a chassis designed for the H100 accepts the H200.
NVSwitch and NVLink
4 chips · 900 GB/s per GPU
Four NVSwitch chips, placed just behind the fans so they receive fresh air first, connect each GPU to all the others over NVLink: 900 GB/s per GPU (7.2 TB/s for all eight), about seven times the bandwidth of a PCIe 5.0 link. This is what makes it possible to spread a large model across the eight GPUs as if they formed a single one. The glowing dots represent these exchanges.
Motherboard tray and CPUs
2 × Intel Xeon or AMD EPYC
Beneath the GPU tray, a 2U drawer that slides out from the back (we have pulled it out to show it). It holds the motherboard and its two CPUs: they run the operating system, load the data and distribute work to the GPUs over PCIe 5.0. They do almost no AI computation themselves: their role is to feed the GPUs without keeping them waiting.
System memory (RAM)
32 DDR5 modules · 2 TB on a DGX H200
The CPUs' memory, separate from the GPUs' HBM: it is used by the system, for loading models and for preparing data. Sixteen modules around each CPU, 32 in all: 2 TB on the DGX H200, up to 4 TB or more from other manufacturers.
Network cards
8 × ConnectX-7 · 400 Gb/s · one per GPU
To train a model across several servers, each GPU has its own 400 Gb/s NVIDIA ConnectX-7 network card, running InfiniBand or Ethernet. On the DGX H200, they are grouped into two modules of four and exit through four OSFP ports at the back; two additional cards are dedicated to storage. The GPUs of two servers can thus exchange data directly, without going through the CPUs (GPUDirect RDMA).
Power supplies
6 × 3 to 3.3 kW · redundant
At the bottom rear, six power supply units of 3 to 3.3 kW, hot-swappable and mounted in redundancy: the failure of one unit does not stop the server. So many are needed because a complete HGX H200 server draws more than 10 kW at full load, 5.6 kW of which goes to the GPUs alone.
Fans
12 dual modules at the front (DGX)
Almost the entire front panel is a wall of fans: twelve hot-swappable dual modules on the DGX H200, around ten from other manufacturers. They draw in cold air at the front, push it through the NVSwitches, the GPUs and the motherboard tray, and exhaust it through the rear grille. On their own, they consume several hundred watts: this is why large installations are moving to liquid cooling.
NVMe SSDs
8 × U.2 at the front (DGX)
Beneath the fans, hot-swappable U.2 NVMe SSDs serve as a fast cache for datasets: eight 3.84 TB drives on the DGX H200, up to sixteen from some manufacturers. The system boots from two mirrored M.2 SSDs inside the chassis. In the center, a small console (USB, VGA) for on-site servicing.
Chassis
8U · 19-inch rack · 75 to 130 kg
The metal enclosure, mounted in a standard 19-inch rack. An HGX H200 server takes up 8 rack units at NVIDIA and Supermicro (6U at Dell) and weighs 75 to 130 kg depending on the model: installing it takes several people and reinforced rails. Behind the GPU tray, a perforated grille lets the hot air out.
Passive heatsink
Copper base · aluminum fins · 0 fans
The H200 SXM has no fan at all: a block of aluminum fins mounted on a copper base (vapor chamber) dissipates up to 700 W. It is the server's fans that push air through the fins; in liquid-cooled HGX H200 systems, this heatsink is replaced by a cold plate.
- Same thermal envelope as the H100: 700 W
- Air or liquid, at the server manufacturer's choice
Thermal interface material (TIM)
Between the chip and the heatsink
A thin layer of paste, pad or liquid metal fills the microscopic irregularities between the top of the chips (GPU and HBM) and the base of the heatsink. Without it, trapped air would insulate the chip and the GPU would lower its clock speed within seconds to avoid overheating.
6 HBM3e memory stacks
141 GB · 4.8 TB/s · 6,144-bit bus
The big new feature of the H200. Six stacks of HBM3e (High Bandwidth Memory) surround the GPU chip: each stack piles up several DRAM dies connected vertically by thousands of micro-vias (TSVs). Six 24 GB stacks = 144 GB physical, of which 141 GB is usable.
- 4.8 TB/s of bandwidth, 1.4 times that of the H100
- Placed a few millimeters from the GPU, on the interposer
- Memory is what limits the inference speed of large language models
GH100 GPU chip (Hopper)
814 mm² · 80 billion transistors · TSMC 4N
The heart of the H200: the same GH100 chip as the H100, manufactured by TSMC on its 4N process. In its SXM version, 132 multiprocessors (SMs) are active, i.e. 16,896 CUDA cores and 528 fourth-generation Tensor Cores, with 50 MB of L2 cache.
- Transformer Engine: FP8 computation for language models
- Up to 3,958 FP8 TFLOPS (with sparsity)
- Can be partitioned into 7 independent MIG instances
Silicon interposer (CoWoS-S)
TSMC 2.5D packaging
A passive silicon plate, with no transistors, on which the GPU and the six HBM stacks sit side by side. It contains the thousands of ultra-fine traces that connect the memory to the GPU: it is what makes the 6,144-bit bus possible. This assembly technique is called CoWoS-S (Chip on Wafer on Substrate); its production capacity is one of the factors limiting the availability of H200s worldwide.
Stiffener frame
Protects the package
A metal frame bonded to the substrate, all around the interposer. It keeps the package from warping under heat and spreads the pressure of the heatsink, which is screwed down with considerable force to ensure good thermal contact.
Package substrate
About 7.5 × 7.5 cm
A very dense multilayer printed circuit board that links the interposer's micro-connections to the solder balls (BGA) that attach the whole assembly to the SXM board. Substrate, interposer, GPU and HBM together form the “package”: the part NVIDIA delivers to its partners.
Power stages (VRM)
12 V → under 1 V · several hundred amps
Dozens of power stages and their inductors convert the 12 V supplied by the HGX board into a voltage below 1 V, but with a current of several hundred amps. They are the module's second-largest source of heat after the chip, hence their position under the heatsink.
Decoupling capacitors
Power supply filtering
Rows of ceramic capacitors around the package, and larger capacitors beneath the board, store and release energy within a few nanoseconds. When the Tensor Cores start a computation, current demand surges: these capacitors are what prevent a voltage drop.
SXM5 board (PCB)
About 14 × 10 cm
The module's printed circuit board. Unlike a graphics card, it has no PCIe connector, no video output and no power socket: everything goes through the two connectors underneath. Four threaded standoffs pass through the board: they hold the heatsink at the right height and pressure on the package, and guide the module when it is mounted on the HGX board.
SXM mezzanine connectors
NVLink 900 GB/s · PCIe 5.0 · power
Two high-density connectors under the board plug into the HGX baseboard. They carry power, the 16 PCIe 5.0 lanes to the CPU and the 18 NVLink links (900 GB/s) to the NVSwitch chips that interconnect the server's 8 GPUs. No cables: this is what makes it possible to stack eight 700 W GPUs in a single chassis.
The GH100 die
814 mm² · 80 billion transistors · TSMC 4N
Seen from above, the GH100 silicon is a floor plan about 28 mm on each side. Everything that computes is in the center, everything that communicates is on the edges: memory on the left and right, NVLink at the top, PCIe at the bottom. On the H200 SXM, 132 of the 144 fabricated multiprocessors are active; the others are disabled at the factory to improve manufacturing yield.
8 compute blocks (GPCs)
8 GPCs × 9 TPCs × 2 SMs = 144 SMs fabricated, 132 active
Computation is organized into 8 Graphics Processing Clusters, each made up of 9 pairs of multiprocessors (TPCs). Each multiprocessor (SM) contains 128 CUDA cores and 4 Tensor Cores: 16,896 CUDA cores and 528 Tensor Cores in total on the H200 SXM. The GigaThread engine distributes tasks across the GPCs; a GPC can also be isolated as a MIG instance.
The SM we are about to open
1 SM out of 132 · next level
A highlighted multiprocessor: this is the one the next level details. Each SM executes groups of 32 threads (“warps”), with its own cores, Tensor Cores, registers and shared memory. Zoom all the way in to enter it.
L2 cache
50 MB in two partitions
A large cache shared by all the SMs, at the center of the chip, in two halves linked by an interconnect network. It keeps the most frequently requested data close at hand to avoid fetching it from HBM, which is slower and more energy-hungry, and it lets the SMs exchange results.
HBM memory controllers
12 × 512-bit controllers = 6,144 bits · 4.8 TB/s
On each edge, three physical interfaces (PHYs) connect the chip to the six HBM3e stacks placed beside it on the interposer, with two memory controllers per stack. The 6,144-bit bus is what gives the H200 its 4.8 TB/s: this is the component that makes the difference compared with the H100.
NVLink 4
18 links × 50 GB/s = 900 GB/s
Eighteen 4th-generation NVLink links along the top edge connect the GPU to the seven other GPUs in an HGX server via the NVSwitch chips. 900 GB/s per GPU, or 7 times PCIe: this is what makes it possible to spread a 400-billion-parameter model across 8 GPUs as if they were a single one.
PCIe 5.0 and GigaThread engine
16 PCIe lanes, 128 GB/s · work distribution, MIG
The bottom edge connects to the server's CPU: sixteen PCIe 5.0 lanes, 128 GB/s in both directions, through which the model, the data and the instructions arrive. Next to it sits the GigaThread engine, the chip's conductor: it receives the tasks (the “kernels”), splits them into thread blocks and dispatches them to the available GPCs; in MIG mode, it isolates up to seven independent 18 GB instances.
A multiprocessor (SM)
1 of 132 · 4 processing blocks
The SM is the basic unit of the GPU: it executes warps of 32 threads on four identical processing blocks, which share 256 KB of fast memory, four texture units and a copy engine (TMA). On a single chip, 132 SMs work in parallel.
Scheduler and registers
32 threads per cycle · 64 KB of registers per block
Each block has its own warp scheduler and dispatch unit: on every cycle, they pick a ready warp (one whose data has arrived) and send its instruction to the cores; while one warp waits for memory, another one computes. Just below, the register file holds the threads' variables: 64 KB per block, 256 KB per SM, 33 MB across the chip, read in a single cycle.
CUDA cores (FP32)
32 per block · 128 per SM · 16,896 per GPU
CUDA cores perform general-purpose 32-bit floating-point calculations: additions, multiplications, comparisons. Thirty-two per block, one for each thread in the warp, all executing the same instruction at the same moment on different data (the SIMT model).
FP64 and INT32 cores
16 FP64 + 16 INT32 per block · 64 + 64 per SM
The 64-bit cores are used for scientific computing (simulation, climate, chemistry), where precision comes first: sixteen per block, compared with just one on a consumer graphics card. The integer units compute addresses, indices and loops while the floating-point cores do the math; keeping them separate allows both to run at the same time.
4th-generation Tensor Core
1 per block · 4 per SM · 528 per GPU
The heart of AI: a unit that multiplies entire matrices in a single instruction, in FP8, FP16, BF16, TF32 or INT8. The Transformer Engine selects the precision on the fly for language models. 4 Tensor Cores per SM, 528 on the H200: up to 3,958 TFLOPS in FP8 with sparsity. Zoom all the way in to see the operation.
LD/ST units and SFUs
8 LD/ST + 4 SFUs per block
The load/store units read and write memory on behalf of the threads; the Special Function Units compute sine, exponential, square root, reciprocal… which are used constantly by the activation functions of neural networks.
L1 caches: data and instructions
256 KB of L1 / shared memory + instruction cache
At the bottom, a fast memory common to the four blocks, split as required between L1 cache and programmable shared memory, where threads in the same block exchange data: 256 KB per SM, just a few cycles away from the cores, and the key to the performance of libraries such as FlashAttention. At the top, the L1 instruction cache holds the running program and feeds the L0 caches of each block.
TMA and texture units
Asynchronous copies HBM → shared memory · 4 texture units
The Tensor Memory Accelerator, new in Hopper, moves large blocks of data from global memory to shared memory in the background while the Tensor Cores compute. The four texture units, inherited from 3D rendering, read and filter images in memory: sampling and interpolation in AI, surface texturing in rendering.
Matrix A: the activations
4 × 8 values in FP8 or FP16
The data being processed: for a language model, for example, the representation of the text at the current step. Each row of A is multiplied by each column of B. In FP8, each value fits in a single byte: this is what doubles the throughput compared with FP16.
Matrix B: the weights
8 × 4 values, loaded from HBM
The model's learned parameters, read from HBM3e in batches. A 70-billion-parameter model means 70 billion values like these, multiplied for every token generated: hence the importance of memory bandwidth.
Result D = A × B + C
16 dot products · 128 multiply-adds in one instruction
Each cell of D is a dot product: a row of A multiplied element by element by a column of B, then summed and added to the accumulator C (the previous result, in FP16 or FP32): this is how large multiplications are split into accumulated tiles without any loss of precision. Beneath the matrices, a grid of hard-wired fused multiply-add circuits performs all the operations at the same time, with data passing directly from a multiplier to the neighboring adder. A 4th-generation Tensor Core chains 512 of these operations per cycle in FP16, 1,024 in FP8; the H200 has 528 of them, running at nearly 2 GHz.
