The H200 is the better inference GPU: the same Hopper compute as the H100 with 141GB of HBM3e at 4.8TB/s against the H100's 80GB at 3.35TB/s, which NVIDIA measures as up to 1.9× the throughput on Llama 2 70B.
Memory only. H200 has 141GB of HBM3e at 4.8TB/s. H100 SXM has 80GB of HBM3 at 3.35TB/s. Compute, NVLink at 900GB/s and the 700W power envelope are the same.
Inference on large models is bound by memory bandwidth, not compute. Each generated token reads the whole model from memory, so bandwidth sets tokens per second, and capacity sets how large a model and how big a batch fit on one GPU.
NVIDIA's published preliminary figures: Llama 2 13B 1.4×, GPT-3 175B 1.6×, Llama 2 70B 1.9× the throughput of H100, and about half the energy per token on Llama 2 70B at the same power.
In 16-bit, about 140GB of weights plus KV cache: two H100s or one H200 at a small batch. For serving thousands of users, an 8-GPU node with tensor parallelism and 8-bit quantisation.
Compute-bound training of models that fit comfortably in 80GB, when H100 is priced well below H200, or when matching an existing H100 cluster.
Tell us what you are running and we reply with a written quote and lead time. No sales call unless you want one.
Figures on this page are indicative, exclude VAT, and were checked on 1 October 2026. Specifications come from the manufacturers' published data. SuperServer.ai is independent of Supermicro and NVIDIA.