# Your Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards — Dan Bjornn, Lease End AI Engineer · 2026-08-20 Dan Bjornn, senior data scientist at Lease End, explains why his team's fine-tuned LLM for customer intent—despite bringing in $12 million at a 50x ROI—became tech debt. The fine-tuning pipeline took a week per retrain, with training itself the shortest step, and each fix caused regressions, so bugs were triaged by tolerable customer pain. He calls this the calcification tax: the model locked them into one provider and an outdated architecture, preventing upgrades. The rebuild swapped the tuned model for skills, prompts, and context on a model-agnostic framework, letting fixes ship in under an hour as uploaded files. Accuracy went up, cost per message rose, but total cost fell. Bjornn concludes fine-tune only when you cannot call a frontier model, and even then the decision must beat the tax. ## Questions this episode answers ### Why is a fine-tuned model tech debt? Dan Bjornn says the more they used the model, the more rigid everything became, which he calls the calcification tax. They were locked into their model provider and architecture, could not adopt newer architectures, and spent all their time keeping the old system alive. That accumulated hidden debt underneath the strong revenue numbers. [9:28](https://aiengineer.podhood.com/6847bec1-5619-44d6-8a5a-617732430710?t=568000) ### What mistakes did the fine-tuned model make in production? Dan Bjornn describes the confused confirmer: after confirming an appointment, a customer replied 'Sounds good' and the model said 'Great, I'm calling youright now.' He also describes the overeager puppy: a customer replied 'Hi, good morning' and the model immediately called. Both happened in production, leading to frustrated customers and missed opportunities. [4:20](https://aiengineer.podhood.com/6847bec1-5619-44d6-8a5a-617732430710?t=260000) ### How did fixing issues change after the rebuild from fine-tuning to skills and context? Dan Bjornn says after the rebuild, finding a problem meant adjusting the system prompt or the affected skill, validating on a curated set, iterating a few times, then deploying by uploading MD files to an S3 bucket. They reduced the process from about a week to less than an hour, while accuracy went up. [12:44](https://aiengineer.podhood.com/6847bec1-5619-44d6-8a5a-617732430710?t=764000) ### Why did Dan Bjornn initially decide to fine-tune the model? Dan Bjornn gives four reasons: they needed better accuracy, fine-tuning would let them use smaller models to lower cost and latency, the narrow structured intent-classification task suited supervised fine-tuning, and he believed having training data would give them vendor control and model-agnostic portability. He later says each reason was beaten by the rebuild. [1:52](https://aiengineer.podhood.com/6847bec1-5619-44d6-8a5a-617732430710?t=112000) ## Key moments - **[0:00] Intro** - [1:52] Dan Bjornn chose fine-tuning for Lease End's intent classifier, expecting better accuracy, lower cost/latency, and model-provider freedom. - **[1:54] Why fine-tune** - **[3:32] Pipeline and ROI** - [3:35] Lease End's fine-tuned messaging app generated $12 million revenue at a 50x ROI within a year, says Dan Bjornn. - **[4:24] Failure modes** - [4:44] The 'confused confirmer' failure: a customer's 'Sounds good' to an appointment made Lease End's LLM start calling immediately, says Dan Bjornn. - [5:54] The 'overeager puppy' failure: a 'Hi, good morning' greeting made Lease End's LLM reply 'I'm giving you a call' in production, says Dan Bjornn. - **[6:04] Retrain cycle** - [7:56] Dan Bjornn says Lease End's fine-tune retraining cycle took about a week, with actual training only an hour and regressions causing whack-a-mole. - **[8:39] Triaging bugs** - [9:28] Dan Bjornn calls fine-tuning's hidden cost the 'calcification tax': the more the model was used, the more rigid everything became. - **[9:31] Calcification tax** - **[11:18] Aha moment** - [11:34] Dan Bjornn's aha moment with Claude Code — changing skills and context, not models — led Lease End to rebuild its messaging app on a model-agnostic agentic framework. - **[12:14] The rebuild** - **[13:08] Faster fixes** - [13:34] Dan Bjornn's rebuild raised per-message API cost but improved accuracy, cut fix time from days to minutes, and lowered total cost. - [14:50] Dan Bjornn asks: 'Can you cross your reason off of this list?' before fine-tuning, citing accuracy, cost, latency, narrow tasks, and vendor control. - **[14:54] Before you fine-tune** - [15:58] Dan Bjornn advises fine-tuning only when you literally cannot call a frontier model, and even then the decision must beat the calcification tax. ## Speakers - **Dan Bjornn** (guest) ## Topics Fine-Tuning, AI Strategy ## Mentioned Anthropic (company), Lease End (company), OpenAI (company), Claude Code (product) ## Transcript ### Intro **Dan Bjornn** [0:14] Alright, hello everybody. Thank you for coming. I'm Dan Bjornn, I'm a senior data scientist at Lease End. Lease End, we connect people who are coming to the end of their auto lease with financing options so that they can buy out their lease and keep their car. Now, as part of this, we built an LLM-based application in late 2024 to help our customers connect with, uh, with our sales team. This application allowed them to send messages through text, they could ask questions about the sales process, they could schedule calls, they could get reminders, all of this stuff. Our first solution used a workflow-based approach, built on top of a RAG system where we searched a vector database of messages that we had already seen and classified with the customer's intent. So, for example, a message saying "call me tomorrow" would be classified as "the customer wants to, uh, wants to talk later." A message saying "I've got time now" would be classified as "the customer wants to talkright now." This has worked, but not super amazing. There's a lot of nuance in messages and conversation, and this RAG approach just couldn't quite pick up on that nuance, and so we started to look for, uh, for new options to improve this. Now, naturally, being a data scientist, my first thought was, "Hey, let's start fine-tuning." This seemed like a fun thing to do, and I was sure that this was theright call. There's a few reasons for that. First of all, we needed better accuracy. ### Why fine-tune **Dan Bjornn** [2:08] Our entire system was built upon us getting the user's intent correct: did they want to talk now, did they want to schedule a call, did they want to opt out. All of this hinged on that decision, and so we needed to make sure that we got that first and foremost. Next, we could use smaller models with fine-tuning, and so this would lower the cost and also lower latency. So this was really important for us because we were responding to thousands of messages a day in real time, and so it would help us scale a lot. Then next, like I said, we were classifying the intent of the user, and so this was a very narrow, structured task that we were trying to do, and so it lent itself very nicely to supervised fine-tuning. We would bucket that conversation in one of six different categories, and the model would learn the differences between those. So it seemed like a great option there. Lastly, I believe that this would help us have a little bit more control over our destiny with the model providers. The idea was that we had the data, and all we would need to do is pass that into a new model, go through the fine-tuning process, and we could get similar results no matter what we decided to use. ### Pipeline and ROI **Dan Bjornn** [3:35] So we could be model agnostic. So this was the approach that we took, and I built a pipeline to collect examples, run LLM-as-judge classifications to label our data. I'd manually review that, create holdout sets, go through the fine-tuning process, check my metrics. This was a data scientist's dream. And the numbers sure helped. Within a year, this application had helped us bring in $12 million of revenue at a 50x ROI. It was pretty awesome. But the whole time, it was quietly accumulating debt underneath that we didn't see. So I want to show a couple examples of how this application could get things wrong. First of all, the confused confirmer is a situation where when customers set up an appointment with a sales rep, we send them a confirmation message to let them know that it's been scheduled and give them the details of that. ### Failure modes **Dan Bjornn** [4:44] So a conversation may look like this: we reach out and say, "Hi Tracy, just confirming your Lease End call with your advisor is set for Thursday at 2:00 PM. We'll call you then." Tracy then sends us a message back saying, "Sounds good." And then our LLM responds with, "Great, I'm calling youright now." It's not what we want. We just confirmed an appointment for a following day, and then all of a sudden we start calling them. This led to frustrated customers and some missed opportunities. The next one, I've come to lovingly call the overeager puppy. The conversation looks like this. So first, "Hi James, this is Alex with Lease End, reaching out about your upcoming lease maturity." James then says, "Hi, good morning." And, "Good morning, I'm giving you a call." Just like a puppy that gets so excited that somebody's giving it attention, our model decided to give a callright there. Obviously, this is not what James wanted. This actually did happen in production. Very embarrassing there. But these are a couple examples of where it went wrong. And don't get me wrong, the app did well. The revenue numbers show that it was working, but it could also mess up pretty spectacularly. ### Retrain cycle **Dan Bjornn** [6:18] The big issue wasn't how to fix it, but how to make the fix manageable. The fine-tuning process was pretty complex. First, we needed to gather examples of the problems that we started to see. Then we needed to ask ourselves, do we have enough examples to go through fine-tuning? If not, we synthesized those examples. We passed it through an LLM. It created some possible examples there. We'd have to validate those, which was a very manual process because we wanted to make sure it had the best training data possible. And then once we had enough, we labeled those with the categorization bins, and we validated those through a manual review. Surprisingly, the fine-tuning process was the shortest part of all of this. It normally took about an hour, depending on the size of the data that we had. But we never got it on the first iteration. Normally what happened was we would fine-tune and we'd evaluate this, and we fixed the problem that we were just trying to solve, but then we caused regressions and other things. And so this turned into kind of a whack-a-mole process where we would solve something new, but then other old issues kept popping up that we had to whack down. This whole process took about a week to gather the data, label everything, go through the fine-tuning process, and iterate, and then deploy. So it was costly. Therefore, we needed to triage all of these issues that we ran into. We asked ourselves three questions before we did any retraining: How frequent is the issue? Is it something that customers are seeing every day? Is it one-off? One big exception to this was if it was hurting the customer experience too much. So for example, this would be somebody repeatedly stating what their preference for a call time is, and then the model ignoring that. Another one would be a customer scheduling a call. We tell them that we've scheduled it for them, but we don't return the payload in the proper way, and so the call never gets scheduled, and so we don't follow up with them. ### Triaging bugs **Dan Bjornn** [8:51] So these kinds of things needed to be fixedright away. But before we did that, we asked the last question: Is there anything that we can do in order to prevent a retrain? Can we have some kind of a Band-Aid fix to get out there so we don't have to go through a whole week-long process for one or two issues? And so we ranked our own bugs based on how much customer pain we could tolerate at the moment. So not a great situation to be in with a production system. This led to what I've come to call the calcification tax. The more we used the model, the more rigid everything became. This manifested in a couple different ways. First, we were locked into our model. You remember when I said that fine-tuning would give us more freedom in what model we did? ### Calcification tax **Dan Bjornn** [9:46] That was not the case. Within providers, there's nuance between one model version to another, and so that changes the training data that you need to provide it. Across model providers, it's extremely different. The structure of the data you need to pass to it would be different, the amount of the training data to get good results, the way to interact with the training interface. All of this caused a lot of complexity, and so it was just too costly for us to switch. And so we kept the same model for consistency because we already had a lot to do with each retraining process, and we couldn't afford to upgrade the model. So the other way that this locked in was architecture. We built this app in late 2024 when workflows were kind of the gold standard if you wanted good production results. And the AI world moves very fast, and we couldn't adapt to that because we were so locked into this, just trying to keep it running, and we couldn't take advantage of the new architectures and improve performance that way. So earlier this year, I had an aha moment. We started using Claude Code for our coding tasks, and I noticed that we never needed to change the model depending on what task we're using. We just changed the skill, the resources that we passed it, the context. ### Aha moment **Dan Bjornn** [11:34] You drop in the better context, you get better results. And I thought, "Why can't we do this with our messaging app?" This was obviously difficult for me to admit because I was the champion for fine-tuning, and luckily we were able to piggyback on a project that was already happening, and so we migrated our workflow approach to a series of skills, tools, and resources that the skills could load into or load up and get that context. And so we pushed this as one of our first production tests of our new agentic framework that was being built already. Now, I want to compare the process before and after our rebuild. Before, we already went through kind of the training cycle, but there was this triage cycle beforehand where we needed to make sure that we had reached a critical mass of problems before we would even attempt to fine-tune again to improve everything. ### The rebuild **Dan Bjornn** [12:44] Like I said, this took about a week, so it was a long process, costly. After the rebuild, it was a simple process of you find a problem, you adjust the system prompt or the skill that was affected, we validated performance on a curated set that we had been collecting over the time that this was in production. We iterate a few times, and then we deploy that simply by uploading MD files to an S3 bucket. This whole process, from discovering a problem to deploying the fix, we reduced down to less than an hour. So it extremely improved all of this, and we could be far more reactive, give our customers way better performance or better experience there. ### Faster fixes **Dan Bjornn** [13:34] Now, I'll be honest, it did cost us a little bit more per message. We were using better models, so the API costs were a little higher. But accuracy went way up. I said before that accuracy was the key to getting all of thisright, and we did that. Accuracy was far better with this than it ever was with fine-tuning. Next, like I said, we reduced our fix process from days down to minutes. Next, we were able to unfreeze our model and finally get that freedom from a vendor that we never had with fine-tuning. Our agentic framework was built model agnostic, so we could use OpenAI, we can use Anthropic, we can use any other model that we want. The important part is the context that we're providing to that model. And then lastly, while it cost us a little more per message, the total cost went down because we were spending far less time trying to keep it up and running and fine-tuning to keep it working properly. So before you fine-tune, I'd ask you, "Can you cross your reason off of this list?" So I thought we would get better accuracy. The rebuild beat the fine-tuned model. I thought we would get lower cost of the volume we were doing. ### Before you fine-tune **Dan Bjornn** [15:08] I was looking at the wrong costs. We paid more per message, but the total cost ended up going down with our rebuild. Lower latency. We did see marginal gains on these smaller models, but they were so small that in practice it really didn't make any difference. And then maybe you've got a narrow or structured task. Our textbook case still became tech debt. And lastly, vendor control. It's not as simple as just plugging the data in. The other two situations where you might have privacy and data control, or you need some offline solution, I would say these are the situations where a fine-tuned model may be useful, but you need to be cautious. There are other solutions out there, but you need to make sure that it's not causing issues in the long run. So finally, fine-tune only when you literally cannot call a frontier model, and even then, your decision still has to beat the tax. Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.