GPU sizing guide
Size GPUs for training, inference, and HPC
Practical sizing for LLM training, fine-tuning, inference, and HPC—memory, node count, fabric, and facility power from Fremont architects.
Sizing guide
Start from workload parameters—not brochure TFLOPS
Document model family and parameter scale, training vs inference, framework (PyTorch, JAX, vLLM, TensorRT-LLM), checkpoint size, rack kW, cooling posture, and growth horizon before locking silicon.
Large-model training typically needs 80GB+ HBM per GPU and multi-node high-bandwidth fabric. Fine-tuning often fits 4–8 GPUs per node with a fast NVMe checkpoint tier. Inference may reduce per-GPU memory with FP8/FP4 and tensor parallelism.
Facility limits—air vs DLC, depth clearance, and power—gate form factor as much as GPU class. NTS flags those gates in architecture review.
Sizing paths
Workload class, platforms, and next steps
Move from RFQ inputs into HGX, PCIe, cooling, and contract-ready quoting.
- Parameter scale
- Latency / batch SLA
- Facility limits
- 80GB+ HBM class
- Multi-node fabric
- B200 / B300 paths
- Departmental training
- Inference density
- Watch rack kW
- Facility plant first
- Clock stability
- CDU sizing
- Cooling choice
- TCO drivers
- CLIN quotes
- Design inputs
- Fabric attach
- Acceptance
- Cluster patterns
- Storage attach
- Facility planning
- Validated BOM
- Fabric sketch
- Contract packaging
Frequently asked questions
Large-model training typically needs 80GB+ HBM per GPU with multi-node fabric. Fine-tuning often fits 4–8 GPUs per node. Inference may reduce per-GPU memory with FP8/FP4 and tensor parallelism—NTS maps this to your framework and SLA.
- Training vs inference first
- Checkpoint tier matters
- HPC sized separately from LLM stacks
2U 4-GPU suits departmental training and edge inference. 4U 8-GPU raises density but watch rack kW and cooling. 8U HGX maximizes scale-out training and requires a facility readiness review.
- Lab / pilot → 2U
- Dense training → 4U
- Scale-out NVLink → HGX
Model family and parameter scale, training vs inference with batch/latency SLA, framework, checkpoint size and ingest rate, rack kW and cooling posture, target node count, and 6–18 month growth horizon.
- Workload parameters
- Facility gates
- Growth horizon
Yes. Validated node counts and fabric sketches quote on SEWP V, ITES-4H, GSA MAS, and SLED cooperatives with CLIN-aligned attachments.
- Architecture review first
- Multi-vehicle packaging
- Integration CLINs available
