To run a 70B model in 16-bit you need about 140GB of GPU memory for the weights plus room for the KV cache: two NVIDIA H100 80GB GPUs, or one H200 141GB at a small batch. At 8-bit it fits in roughly 70GB, and at 4-bit in about 35GB.
Parameters multiplied by bytes per parameter. 70 billion parameters is about 140GB at 16-bit (FP16/BF16), 70GB at 8-bit (FP8/INT8) and 35GB at 4-bit.
The KV cache holds attention state for every active conversation and grows with context length and concurrent users. Fifty users at 32,000 tokens of context can need more memory than the weights themselves, which is why production serving uses multi-GPU nodes with NVLink.
One H200 serves a 70B model to a handful of users. An 8-GPU HGX H200 node with tensor parallelism and 8-bit weights serves thousands of concurrent requests. Capacity should be confirmed by a load test on the real workload, not estimated from parameter counts.
Full fine-tuning needs more than 560GB of GPU memory, which means a multi-node cluster. Parameter-efficient methods such as QLoRA freeze the base model in 4-bit and train small adapters, bringing a 70B fine-tune to about 40GB, within reach of a single node.
For a 70B model in production, an 8-GPU HGX H200 server such as the Supermicro SYS-821GE-TNHR or AS-8125GS-TNHR. For a single developer, a 128GB unified-memory desktop system runs 70B at 4-bit for one user.
Tell us what you are running and we reply with a written quote and lead time. No sales call unless you want one.
Figures on this page are indicative, exclude VAT, and were checked on 1 October 2026. Specifications come from the manufacturers' published data. SuperServer.ai is independent of Supermicro and NVIDIA.