The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside
Jul 26, 2026 · 17:31
Marah Abdin and Robert McHardy from poolside detail their synthetic data pipeline and pre-training tribulations at scale, culminating in a new 118-billion-parameter model for agentic coding that outperforms competitors. Marah describes using synthetic data to rephrase content and fill gaps, with a configurable pipeline (Hive) involving agents, orchestrators, and supervisors, covering rephrasing, multistage workflows, cross-domain porting, and multi-turn chats. Robert recounts failures like broken GPUs causing data corruption, a BF16 accumulation bug that stalled training, and a race condition in FP8 kernels silently corrupting 0.5% of gradients, all caught by model replica hashing. Their Laguna S model (118B total, 8B active) trained on 30 trillion tokens across 4,000 GPUs beats GLM 4.5 Air and other models on coding benchmarks like BigCodeBench and SpeedBench agentless, while remaining competitive on general knowledge.
