Infra behind Krea 2: How to train and serve at scale — Gabriel Jorge Menezes, Krea.ai
Aug 18, 2026 · 16:55
Gabriel Jorge Menezes of Krea explains how Krea 2 was trained from scratch on thousands of GPUs, arguing metrics, not mystery-solving, made scale survivable. He calls GPU utilization a lie, tracking tensor core utilization instead, which climbs as resolution steps from 128 to 1024. Any GPU above 78 degrees gets pulled without debugging; custom InfiniBand and NVLink error collection caught most failures, which were silent cross-node timeouts. Aggressive checkpointing on a Wekker filesystem writing nearly a terabyte per second let crashes restart and run 24 hours on same nodes. Production and training share one cluster: gang scheduling kicks inference to external providers via a Virtual Kubelet fake node; taints stop wasted GPUs; a descheduler migrates pods back so the site never drops.
