FlashAttention-3 + new tensor cores deliver 4x training speedup on H200s – backward compatible with all major frameworks.
Your LLM fine-tune that took 48 hours? Make it 12 with one software drop. NVIDIA’s CUDA 13.2 just redefined what’s possible on existing GPUs.
Announced today, CUDA 13.2 integrates FlashAttention-3 kernel (2.5x faster than v2), FP8 tensor cores for Hopper GPUs, and automatic kernel fusion. Result: 4x end-to-end speedup training 70B models on H200s, 2.5x on A100s. PyTorch, JAX, and TensorFlow all supported out-of-box.
This hits straight at the dev pain point: sky-high compute costs. Fine-tune Llama-3 on your dataset for pennies instead of AWS bills. Multi-node scaling improvements mean 8x H100 clusters hit 95% MFU – efficiency rivals supercomputers.
AMD’s ROCm 6.1 trails by 30% on attention kernels; Intel Gaudi3 lacks ecosystem maturity. CUDA’s moat? 90% market share and seamless upgrades. No framework rewrites needed.
Download from developer.nvidia.com/cuda-13.2 today – works on Ubuntu 22.04+. Run nvidia-smi post-install and watch utilization soar. What’s the first model you’re retraining?
Source: NVIDIA Developer Blog