Your Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards — Dan Bjornn, Lease End
Aug 20, 2026 · 16:39
Dan Bjornn, senior data scientist at Lease End, explains why his team's fine-tuned LLM for customer intent—despite bringing in $12 million at a 50x ROI—became tech debt. The fine-tuning pipeline took a week per retrain, with training itself the shortest step, and each fix caused regressions, so bugs were triaged by tolerable customer pain. He calls this the calcification tax: the model locked them into one provider and an outdated architecture, preventing upgrades. The rebuild swapped the tuned model for skills, prompts, and context on a model-agnostic framework, letting fixes ship in under an hour as uploaded files. Accuracy went up, cost per message rose, but total cost fell. Bjornn concludes fine-tune only when you cannot call a frontier model, and even then the decision must beat the tax.
