The State of Model Routing — NVIDIA, Cognition, OpenRouter

Those have been really exciting. We've tried to get a bunch of the industry leaders together to talk about some of the problems that we're facing as we try to run more on local. If you guys were here for the first panel, one of the things that we talked about was model routing.

We firmly believe that we're in a multimodal world. I think you heard this from many of the panelists. Anyone who is deploying AI in production and who is doing so locally is seeing that multimodal world. That's why we released these Nemotron models at NVIDIA.

Everything is released from the datasets to the weights, with recipes so that you can customize them. We do that because we know that people customizing models is going to be huge. And so this panel is really exciting because we're going to talk specifically about model routing.

So as you are picking which model to use, how does that essentially, how does that tooling itself look? Do you guys want to introduce yourselves?

Yeah, sure. I'm Walden. I'm the co-founder of Cognition. We build Devin, AI software engineer. In addition to the product, we spend a lot of time partnering with our customers to figure out how they should deploy these models and these agents.

And one of the things they're constantly asking us nowadays is basically, how do I know the ROI of our models, and how do I know which tasks I can actually let our engineers spend the most expensive models on versus letting them use a more cost-efficient model?

And so that's why we're also thinking a lot more about multimodal routing nowadays.

Totally.

I'm Carter. You guys heard from me a little bit earlier, but if you weren't here, I'm a developer tech engineer at NVIDIA. And ultimately, I spend a lot of time thinking about how to get intelligence into as many developers' hands as possible.

And something that is continually becoming not an issue, but something that is top of mind for a lot of developers is, as you use more intelligence and the frontier models get more expensive, it becomes somewhat cost prohibitive to use the best tools, what feels like the best tools, as much as you would like to use them.

And so this has become a recent focus is, how can we still get the same desired outputs, but actually both as an individual developer, but also imagine startups and small companies, how can you leverage this incredible tool without totally breaking the bank?

I'm Tanay. I work on model evaluations, both in terms of its accuracies and efficiency and cost understanding of the model. And then I try and understand those, implement those learnings, and help build a router. So it's basically my job is to understand the behavior of the model on an intimate level and then use those learnings to both improve the model and try and design a system of models that can work together with each other.

Totally. Yeah, I love a lot of the research that you're doing at NVIDIA as we kind of see this space through. I think what's really interesting is model routing itself is pretty new still. And so what you'll notice is there isn't a very clear solution here.

That was something that came up on the first panel, is that there is a lot of space for startups and for companies in the ecosystem to fill in a solution here, because we're still figuring out how to best do these patterns.

And I think, Walden, I want to kind of ask you. So Cognition just released Fusion, your guys' model router. And when you guys released it, you in your blog said that you're actually getting better performance than Fable, than these frontier models.

And I feel like that was a very surprising statement to hear, because we were thinking that you're getting as good or close enough, usually, when we're running on edge, when we're running local, in these compute strains, smaller footprint models.

But you guys are getting better. Can you explain how?

Yeah, absolutely. So I also want to be clear about something here is, we're not saying that we gap above Fable level performance.

Totally.

In the same way that maybe Fable level performance gaps above other models. I think, actually, there's this really unintuitive dynamic where smarter models actually get better and better at delegating work. And so one of the philosophies we had with building a model router is, we don't want to route people to a dumber model.

And then suddenly you're stuck with a model that doesn't know how to do your task. And next thing you know, you're switching yourself back to a smarter model anyways and now taking that expensive cost. In general, we think a lot of the existing model routing systems out there are probably the same ones people have been using like a year ago.

And so we really wanted to put out a new framework that actually lets people still feel like and still have a frontier model in their system while getting all these cost benefits. So yeah, we're reducing the cost of Fable level intelligence by 40%.

The way we do that is we allow Fable to still do the planning and the hard decision making, but delegate a lot of the work to an implementation model. And the implementation model can be one of these open source models, be it a cheaper mini model.

The unintuitive thing is, even though it's cheaper, because you're delegating the work to another model, you can let that model go at the task with much more depth and intensity than you might otherwise. You can spin off three sub agents to go and explore the codebase.

And maybe that's actually more comprehensive than if you had just let Fable explore the codebase itself. So you're actually getting this nice trade off where it's both more cost efficient and also more comprehensive overall.

Interesting. I see. So you're saying by using a bunch of smaller models, you're essentially, for one example, scouring the codebase. You can explore it potentially better than if you were to just have one model, I don't know, figure out with what it's limited, with its limited context, with whatever path it's on.

Yeah, totally.

But also, if you think about the budget of if you were to say the frontier model costs this amount per token and the smaller model is this amount per token and it's significantly cheaper, then you can use a lot more tokens from the smaller model.

That's fair. I see.

Still within the budget that it would have been from the frontier model.

I would also like to encourage everyone to think there are jagged capabilities in most models. So coding is not one domain. Within, let's say, data visualization, there'll be scikit-learn, there'll be matplotlib, there'll be something else. It largely comes down to the training corpora that went into each of the models.

So one model, while you're trying to do X type of work, let's say data visualization, and the other type is Y, that means, let's say, model building. Let's say you're trying to have a data science workstream where you're trying to optimize for some kind of prediction and then visualizing your results.

Within that task, different models will have different strengths. So it's not necessary that model A, if it scores higher on a coding benchmark, is just plain better at every task that is. So routing is a task of intimately understanding the behavior and strengths and weaknesses of different models and then applying them thusly.

I would encourage everyone to think, hey, models are strong at different things rather than there's one model to rule them all.

I see. And by the way, real quick, thank you, Alex, for joining.

Sorry I'm late.

No. Oh, is that still?

I might need yours.

Sorry I'm late. I'm Alex from OpenRouter. Thanks for having me on.

Yeah, of course. Thank you so much. You cameright from the airport. So this is perfect. I think, Tanay, that's super interesting. So the way that you're thinking through model routing, it's not even just delegating to necessarily a smaller model.

And maybe this is kind of what you're saying is, can you put essentially a swarm of agents to accomplish the same task? And suddenly, routing the task between them is a problem to solve in and of itself.

Yeah. So if you look at, let's say, let's take an easy example. Let's take science or scientific discovery as an example. Usually, these are one-shot problems. It's incredibly hard. You have models thinking through this process. So in that, you have tons of subdomains, like tons and tons and tons.

So in that aspect, if you think about post-training, like the post-training process of a model, they'd be tuned with different teachers. They'd be tuned on different subtasks. So

those overlapping strengths will be readily apparent when you try and understand failures of each model on different, different subtasks. Once you understand that, you can orchestrate your system to leverage that arbitrage essentially. And that essentially becomes free. So I think this is on LM router bench.

There are tons of benchmarks out there. But if you use these techniques, you can get up to 10% higher accuracy even. It depends on the model pool, depends on the task at hand. But I would encourage to think about the complementary nature of models.

I see. Do you see so in kind of the way that you were describing the way the task gets broken up, do you see that some of the smaller models, because the token cost is cheaper, are they using more tokens?

Are you specifically routing so that they do, so that they are shadier?

Oh, yeah. They absolutely do use more tokens. I actually want to kind of riff on something that Tanay was saying, which is, a lot of times when you look at these different benchmarks, you'll see that the small models will perform better than even the frontier models in certain cases.

I think a lot of people, they look at this and they immediately jump to, oh, how can we just route the task where the smaller models do better just straight to the smaller models? I think that one of the themes we really want to emphasize with our recent blog post and release of Devin Fusion was that this kind of naive initial routing based on the task type is extremely fragile, especially the more agentic the task you work on is.

So for example, a real developer, you might ask your agent first, oh, how does this codebase work? And then you go deeper and you're like, OK, actually, can you implement some features for me? And then you go deeper and it's like, oh, can you now go do a live test of this feature and debug deep cases?

The complexity changes and the type of task changes over time. And you don't want to be left with some subpar model for the task that you're now on. I think this is why people like frontier models so much is they're just generally intelligent and they're capable of shifting between various different domains, even if you can eke out better performance in very specific tasks.

And so the challenge is, how do you get a small model to know that it's out of its depth and you need to now go switch to another model or go to a smarter model? And our solution to this is you kind of just always have this main frontier agent that's watching.

Even if it's not the one doing the work, it should at least be keeping tabs and figuring out, OK, wait, the agent I delegated to now is out of its depth. I need to kind of move it to something else.

And overall, just the guarantee of always having frontier intelligence present, I think, reduces the fragility of these systems quite a lot.

How does the sharing of context between one of those smaller agents who has basically completed up to some level of a task and decides, actually, I don't think I'm theright person for this. I need to hand it back to the foundational model?

Of course, you don't want to have the entire trace of that smaller agent be passed back to the larger model. So how do you get that level of specificity while basically providing the information it needs, but not more?

Yeah, absolutely. So I think the context here is, it's actually very easy to actually create a system that's more expensive as soon as you're running multiple models together. Because, oh, no, this one file read now, every one of these models is now reading this one file reading.

So you're doing charge like three times as much. The trick that we spend a lot of our time on is, most of the context by default will only be going to one model. So most of the context, let's say, will be going to the small model.

But the thing you need to then tune very well is, OK, maybe you still show what files it's reading. Maybe you show the high-level thinking of what it's doing back to the main model. Maybe you have the small model, you tune its ability to present the context back to the main model.

And actually, a lot of these problems already have been well studied in many domains already. Context compaction is something you already have to solve if you want to do really long-running agents. And so this problem of taking long context, compacting it in a way that is now understandable, is one that you can also apply to this domain and just kind of give the compacted context back to the main agent.

Context compaction is something that I'm familiar with, but I hadn't really thought about as you're doing model routing and as you're trying to share context across now potentially many models, you're expanding the amount of what could be seen as wasteful tokens or redundant tokens just because you have to process that across the many models.

Yeah. Yeah. I think the way I describe it is, I think we are early in the model routing domain.

Totally.

I hope that a year from now, even the techniques we kind of used for Devin Fusion, you people look back on that and are like, oh, these are some really legacy ideas. And now we have much better methods at routing between models.

And when people actually start co-designing their models with this in mind, we're going to be in a much better world.

Yeah. I echo what you say. I think routing will evolve as the task evolves when you start the task. So it's more useful to see things in terms of subtasks and sessions than individual problems that you're trying to solve.

Because more than likely, when you're working through a problem, you're asking a lot of questions. You're exploring different things. And it is imperative that you try people who design routers, it is imperative that they try and understand these phases of different complexities and then try and apply some logic for essentially sidekicking tasks or leveraging expertise from other models.

That's pretty on point.

Yeah, yeah. I'd love to hear from the router guy.

Yeah. I think these are important points. And one of the biggest debates I think we have internally is whether that outer model that's doing the orchestration should be the big model or the small model. You get very different results depending on your choice.

And it's not even clear what the pricing impact would be. Because if your outer model that's doing the orchestration is the big model, it can leverage its caching to make more of its decisions. And its caching is going to be a dramatic price savings compared to the small model's caching a lot of the time, especially for

issues that are on the bright line. Zooming out a little bit, I think what you want from all the models out there when you do model fusion is to benefit from all the data that is being trained on across all the labs and not just the data from one lab or one source.

And a model is just like a combination of the data and its understanding of the data, both its compute and the quality of its RL. So

long term, I think you want models where they know that, oh, this is in distribution. This is in my data. You can use small models pretty easily and get a cost savings. But if it's out of distribution, small models may actually increase your cost because of how often they'll call tools and how crazy their loops will be.

If you run terminal bench on Opus and Haiku, Opus will do about three times better at 1/10 the cost of Haiku, even though Haiku is significantly cheaper per token. So it really becomes a huge problem if you use a too small of a model, particularly on tasks that are out of domain for the training data.

When you're doing something like classifying text, like, hey, is this a person's name or is this an organization's name? That's super in domain. So you don't want that kind of task to go to a large model. You want it to go to a small model.

Everyone has that in their domain. So being able to understand in domain, out of domain is a lot of work that we're doing for OpenRouter Fusion. And then also figuring out how to orchestrate the outer and inner models for different types of tasks.

And

it's an early industry. It's like an early field of research. Most research on model fusion has not been very detailed, not been very optimistic sometimes. It's only just recently getting more optimistic. And I think I'm personally very optimistic about it.

And

we're a very ecosystem-driven collaborative company. And a lot of our we work with a lot of partners to try to help improve their orchestration pipelines with good primitives like the subagent and the advisor tool, which is kind of similar to what you were talking about.

I'm curious. So help me understand. It makes total sense that a small model, if it's in domain, would be cheaper. But if it's not, then it's going to thrash around as it tries to get an answer. When you're describing whether

the main agent should be the local model or the cloud model, is that a decision that's then dependent on whether the task is something that's in domain or not? Does my question kind of make sense?

I don't know. Basically, it's kind of early

to say. I think

the results that we published, which are a couple of weeks ago, which were focused on deep research, not coding, we had the smart model be the wrapper model, be the outer model. And we got the best results from doing that.

But for deep research, it works the best. For other tasks, it's kind of unclear. Fusion is not super well optimized for coding. And it might be that a smaller model ends up being a higher efficiency or fewer dollars per successfully completed task.

But it's kind of early to say.

One thing you said earlier is, oh, you get the caching benefit from the main line agent. You actually can get the passion benefit from the side agent. And this is actually one of the key things we talked about with our Devin Fusion launch is that you kind of are leaving a lot on the table if you do a main agent and subagents type system.

So we don't use subagents. We use what we call a sidekick, which is one subagent that continually has a running context. So the main agent doesn't need to reprovide context from earlier. It's all still in the KV cache.

It's 10 times cheaper on all those cached tokens. And then if you want to switch the smart model to be the one on the side or the one in charge, it's actually totally fine. And you can kind of do the swapping back and forth.

We're also spending a lot of timeright now thinking about how do you train models to actually work collaboratively with other models. I think there's a lot of literature out there on how you RL one model to do a task end to end.

How can you RL a model to also be good at collaboration? And when we think about it, we actually tried both of these setups where let's RL the model being the orchestrator and the one deciding what gets delegated to other models, see how well that performs.

And we also orchestrate it in a way where the model we're training is actually the executor, the sidekick, and see how well is it at executing other models' instructions. And we expect that to be probably a big lift in this next step of multimodal orchestration is don't just take models as they are and orchestrate them, but can you actually co-design your models with the orchestration system?

Yeah, that makes sense. I mean, with Nemotron and with all the foundational models, we're essentially post-training them for the harnesses that they're getting used. If the harness is going to include a lot of routing, then that makes sense.

That makes its way into the post-training.

Yeah. Are you guys thinking a lot about kind of model training at NVIDIA for these kind of purposes?

Yeah. So we have a technological.

The mics go?

Testing. Oh, this one works.

OK. So we have a technological flex run. So we have a setup where there's a main model. Then we distill it into smaller footprints. And then based on the task at hand, you can switch which model does the decoding.

So there's a lot of fancy stuff you can do within a model artifact to essentially only activate a class of model or a section of weights depending on the task at hand or the complexity at hand. In most cases, you can essentially understand the novelty of a question to a model if you have access to the recipe with which it was trained.

So this works very well for open models or any model you have access to its data for. Because you can literally decide if it's in see if it's in distribution or not. Again, if you have studies from when it was trained, you can also see how much essentially how much was your distillation gap across teachers and the artifact that you trained.

Because sure, you have domain data from all the different domains you're tuning, but it's not guaranteed that it absorbs all that data evenly across the model. So

it becomes very interesting to start thinking about these flexible weights and flexible model sizes, essentially. I wanted to add about the context piece. So how do you think about ASTs and context compression representations? Compaction, in its very nature, is lossy.

So just like headroom is there, RDK is there. These code bases are usually designed to have representations that we carry forward through life. And you essentially give models the capability to further expand on them. It's more like loss-less-ish compression, which can retain states of models or states of agents.

What do you think about that?

Yeah. I think this gets to kind of a fundamental philosophy of how agents and context should work. One exercise I like to do is, as a human, how many numbers can you if I just start spitting out numbers now, how many can you remember before you start losing track of them?

I think it's actually very few. So in some ways, you could argue that your context window is actually shorter than these language models. And yet, you can actually be very effective at that. Your context is very lossy. I think one of the nice things that people are starting to realize with agents is you have a lot of non-lossy systems that you can fall back to.

So you have a file system. If in your memory all you remember is that you read some file earlier, you don't need to remember the whole file. You maybe remember the important parts, but you can still have the full version of the file on your system.

And that's kind of my goal when I'm thinking about how do we build a good context-engineered harness is the harness should have everything it needs to find what it needs to have, even if it doesn't have everything immediately available.

In that case, do you think that the context sharing problem will become cheaper and less of a problem in the future?

Yeah, it's definitely possible as well. I've seen cases where the sidekick agent does a bunch of work. It tells the main model, oh, yeah, here's all the things I found. Instead of dumping the full thing, it just references them by file.

And then the main model is actually generally you find these larger, smarter models. They're actually more token efficient with how they use tools and how they read. And so they actually read the files in a way where they only see the important parts.

Or they decide that, oh, actually, I only need to look at a subset of this. Or, oh, I can run a single command and just know if everything is done properly. It's actually quite amazing the fact that these multimodal systems, they actually seem to scale and get better with intelligence, which is not something we should just take for granted.

It's not obvious that actually more expensive models are actually creating an overall cheaper system.

Yeah. The scaling laws, if you have a larger model, it's going to be more efficient with its tokens. Smaller models, less efficient with its tokens. I guess I had a question for you, Alex. Do you guys weigh more importance on the actual infrastructure side of routing?

So for instance, KV cache-aware routing, or is that where most of the business isright now? Or are you seeing strong pull of people actually deploying routers in production?

OpenRouter is like a marketplace for language models. We exist at we can't see into the KV caches of models unless we're running them ourselves, which is pretty rare. We do spend a lot of time optimizing for cache hits.

And we pass through cache hits directly to users. But in terms of KV cache optimizations, we can't do any specific work there. What we do for model routing is we try to find the best model or best combination of models for the prompt.

And then when we see a cache hit, we'll use up the duration of the cache and send the downstream customer the full savings of the cache hit. There's more work that we can do here where we could say, OK, this looks like something where there's significant benefit to switching the modelright now, but you haven't used up the full cache.

You still have like two minutes left. And we think it's probably worth switching the model and losing the rest of your cache and letting people kind of tweak their tolerance for that behavior. We've been doing a little bit of that, but we haven't exposed it to customers yet.

What is next for you guys in terms of your model routing? Because as you mentioned, it is kind of like a different direction from the marketplace business that exists today. So I'd love to hear.

So we've been doing we've had an auto router for like two years almost. But when we launched it, there was no adoption of it. People really wanted to use specific models. And the auto router just had no real usage.

We mostly saw it as like a discovery playpoint. Like, hey, this is how you discover which model might be good for your prompt. And then

around January this year with OpenClaw, it exploded. And the reason it exploded is because there's this fundamental idiosyncrasy in OpenClaw where it sends heartbeats every 10 minutes to your model of choice just to see if the client was still active.

And that means that if you set Opus to be your default model, it would be using a lot of tokens on this heartbeat process. And so this was the very beginning of a very popular app with two completely different intelligence needs, completely different.

And the open source models have improved to a point where it makes sense to segment the market in at least those two areas. And so that was how it got started. And then we saw a lot more segmentation blossom afterwards.

And now a whole bunch of agents and apps on OpenRouter use the different routers that we have. And we have a couple of them. We have Pareto code, which gives you the Pareto optimal model for coding tasks given a certain threshold that you can tune.

We have Fusion, which orchestrates multiple models and gives you a Fusion result. And then we'll have other experiments in the future. What we want to do is basically create good primitives that developers can use to get really advanced with how they use model orchestration, kind of like Sidecar, sort of like that, but also give people a really easy thing that they can just set a slug to that works with all harnesses and just gets the job done.

I feel like it's super interesting how much of a perfect storm there is for model routingright now. Because on one hand, ignore agents, ignore OpenClaw for a second. Just to squeeze better performance, it seems like we should be smarter about how we tackle problems.

That's obvious. If you make a plan, if you make a strategy, that's a better way to go about your day. So I'm not surprised that you're going to see better code get written or more performant code get written, less buggy code get written if you break the problem down.

And so routing specifically for that use case makes a ton of sense. But then hearing this, yeah, the profile of workloads changed with agents. It went from I ask questions, I get a response. Then it went to reasoning, where I ask questions, it reasons, and then it comes back.

And then it went to, yeah, this heartbeat. If my agents are running optimally, there's a token being generated every second. And suddenly, that is its own need for model routing. And it feels like hearing the different solutions to kind of tackle each of those is very interesting.

Well, I was just going to say, I think that, yeah, the use cases for model routing, there are many of them. And so one could be getting a better answer. One could be saving money and trying to get the same answer.

One that we haven't even talked about yet, which is probably the most relevant, maybe even to this crowd, is when do you want to actually run a model locally versus when do you actually need something like a frontier model to perform that task?

And that might be something to the effect of for privacy-protecting information. When I'm running some local, can you detect that my prompt has sensitive information? And if so, do that on-device, but then maybe even anonymize some of that information to go do the more advanced workloads on top of that information in the cloud.

Another example would be, again, for the cost savings, where it's like, hey, I bought this DGX Spark, and I know I'm not at 100% utilization. How can I make sure that as part of my workloads, whether it's the heartbeat in OpenClaw or what have you, that I'm leveraging that compute to the fullest of its ability?

Because I'm only paying for the electrons that are coming in from my power bill, but I'm paying full price for the tokens in the cloud. And I think that's a whole other area of model routing that I know that we're doing some work with at NVIDIA that I think will be really cool as it starts to the hybrid of local and cloud starts to really emerge as its own sector.

Yeah. I'd be curious to know what you guys take is on. If you self-host a model, the cost dynamics change. You have a considerably higher cost at a higher context length because your throughput slows down as the context gets deeper.

So rather than switching to a cheaper model, even if you have self-hosted models in data center, you can use compaction to bring your throughput back up. Have you guys thought about this paradigm, like compaction versus just routing? Because one is you have fewer tokens to work with.

One is we have cheaper tokens.

Yeah. I think in practice, compacting alone doesn't solve the cost or throughput problems because a lot of times it's just like the differential in model intelligence and cost is just so big where also, by the way, when you compact, you're taking a cache miss.

So you're actually now paying 10 times as much for those input tokens if you didn't compact. The main reason we compact is actually intelligence. All these modifiers, they advertise some insane context window of like a million tokens. I would never recommend using these models past 200K tokens.

Under 100K, if you can, the intelligence just kind of falls off a cliff at some point. Sorry, Anthropic, if you're watching. But I think that compaction is a very useful tool if you are going to have to take a cache miss anyway, one way or another, like when you're routing to another model and you want to just minimize the window there.

Do you find that in the Sidecar, when small models are generating lots of tokens, is that one of the best reasons to switch it to a larger model? Basically, when small models generate lots of tokens, I wonder if that's like

the root cause of intelligence problems down the road. You want your big model to generate the big token chunks. The small models to generate smaller token chunks.

And on that question, you mentioned a small model essentially needing to flag that it needs help from the larger model. What is that mechanism? Because that seems like what's the indicator, and then what's the mechanism for it to do so?

Yeah, totally. So there are a lot of mechanisms we talk about in our blog posts about how we just detect that we need to change the model up.

I guess to answer your question first, how does the small model detect? Actually, the thing that we spend a lot of time on is how do we make sure the small model is good at detecting it? Unfortunately, there's a lot of cases where you do need the big model to detect it.

One thing that we don't go into the blog post is you have some kind of cadence on which you're refreshing the cache anyways. Because by default, there's some five-minute lifetime on these caches. If you're going to go refresh a cache anyways, you basically can get a free big frontier model call if you kind of ask theright question.

So it's at that point where you might say, hey, just take a look at what the small model is doing. Does it feel like it's kind of going into some rabbit hole and needs some help now?

What's the need for the five-minute refresh?

It's just like a practical you have to pay some kind of cost to keep these KV caches warm. And so most caches just get evicted on some kind of cadence.

How it works is at inference time, you only have so many caches you can keep loaded in the GPU. So once if a cache is not being used again and again, it's offloaded. So it's lost, essentially. So that's why the inference provider asks you for money.

But if you self-host it, you can get around this problem. You can make it as long as you want based on your business logic.

Do you see a world where we'll have much more dynamic cache durations rather than just the five-minute, one-hour?

It depends on who's deploying the model where. So if you have a GPU which has a lot of memory, which the ratio of, let's say, SMs to memory is memory more heavily skewed, or if you're working with unified memory and you have systems like Vera Rubin, you have a lot of tricks to play here.

The five-minute window is what a lot of providersright now put. But

that's more an

operational determination rather than a science-based or a core physics law determination. So you can technically see over time maybe some APIs are priced differently. But if you self-deploy again, you can get past a lot of this. The cost economics really change when you move from self-hosted models to API providers because you have a lot more control, and you don't have to guess the shape of your workload.

So let's say if your workload is 32K on average, 32K cache, 1K input, 1K output, and someone else is, let's say, 64K, 1K, 1K. If you use some provider, they are amortizing everyone's use case and then giving you a price.

And they have optimized "for general use." If you self-host, you can optimize specifically for your use, and you'll likely pay much less.

Yeah. This is kind of like the level of hardware-software frontier that we kind of like thinking about. When we started Cognition and we were working on the first agents, I think one reason why no one else worked on agents is they were just extremely expensive.

This was before cache tokens was a thing that API providers paid for. If you were sending 100,000 tokens and the same 100,000 tokens, you were paying full price for those tokens back in 2024 when we started. One of the key things that let us build Devin and build these first agents was we actually bought direct compute capacity from these providers.

And instead of paying on a per-token basis, we just paid for the underlying compute, knowing that the economics of the compute was that we were actually paying far less for the cache tokens that we'd send over. And then nowadays, there's similar things I would like about having a version of the cache that maybe you can just back out to storage in S3 or something, and it's like hold for much longer.

Yeah. Now, this is not extremely relevant to a DGX organization setup, but if anyone's looking to do what you guys have wanted to try out, Dynamo, we have a lot of prefix cache optimizations in there.

Yeah. And then going back to your question, Alex, I think you said, oh, when a small model is going off and generating a ton of tokens, is that an interesting time to back off? To be honest, we haven't explored that yet.

So that might actually be a very interesting thing to take a look at. It is weird. I think some small models do tend to kind of be less token efficient than others, but they also seem to be trained on their own traces.

So maybe in a way, it ends up balancing out. A lot of these things, I feel like we have to be very empirical about to actually know.

So just to add on that, you have a lot of these days, there are a lot of hallucination probes, so probes that work on either the internal state of the models directly. So you can have some form of either magnetic analysis done or linear probes or just n types of probes that you can see.

And you can essentially rate how much you think is tending towards hallucination. So that kind of gives you a proxy for how lost it is, how lost a model is in its thinking. So you can use different kinds of probes to understand the perplexity within a model.

That's interesting. So yeah, instead of using the quantity of tokens as indicative of a model being lost, it's hallucinating more.

Yeah. So essentially, what is cache? It's the prefill states. So what is a prefill state? It's just a vector at the end of the day. So you can tune all kinds of classifiers to understand different aspects of those collections of vectors.

So with those kinds of probes, you can guesstimate a lot of states of a model.

I see. One question I have is different models behave differently. And that kind of means that these prompts are importable. So as you're doing model routing, how do you handle essentially, if you're going to a different model architecture, what do you have to do to the prompt, and how much is that a factor into either of your guys' model routing solution?

How is the prompt itself a factor into the routing?

Yeah.

Yeah. Well, I think with building agents, there are all kinds of paper cuts and edge cases that are domain-specific. And the value of an agent company, like the value of Devin, is all these doom loops that you've discovered that are across all industries and the best ways to recover from them.

And it manifests big time in what the prompts are going to be, both for how the advisor model gets called, the smart friend,

how the subtask agents get called. And the best thing is that anyone can, any engineer or any agent can inspect the traces and adjust the prompt and then see the live accuracy long time. So I mean, basically, I just think that that's part the prompt is part of the startup building process and is also really easy to observe and

have multiple people and agents collaborate on them.

Yeah. One thing I'd love to do with our Fusion product, and we don't have this yet, and so this is maybe a preview of some things we work on, is you can tune it against a data set. But the real thing you want when you're building a real agent someone uses is just tune it against what actual people use it for and what actual models they get routed to.

And so there's a lot of signals for this. If someone sends a prompt and then Devin is working, and then you see that the user decides themselves to upgrade to a different model, or they decide to downgrade, or the system detects that we originally sent to the wrong one, we now got to replace, that's actually a really useful stream of signals.

And we're actually getting into this world of auto research where maybe we can just have a constant stream of prompts, what it should have been, what it was instead, and build a system internally that's just capturing all of this and then reiterating on our routing system until it eventually kind of fits the real production data.

That's kind of now that it's public and people are using it, this is now something that we're thinking about.

Have you guys looked into prompt tuning, and do you find it useful? Like, say, Japa?

Yeah. There are these prompt tuning frameworks from a few years ago that tried to do some kind of gradient descent type thing. I'm actually personally less bullish on these kind of low-level mechanical prompt tuning harnesses versus just telling a smart model, here is the decision that was made and the context.

Figure out why it went wrong. Sometimes you can do something as dumb as asking the model, why did you do this instead of this, and cite the prompts. And then just have your agent, your Devin, just go and update the prompts, rerun the test as a regression, make sure it changes.

It's a lot heavier weight of a system, but I kind of trust the intelligence of a system like that a lot more.

So we're running out of time, so we're going to wrap up real quick. But I think what's really interesting is just from talking to you guys, we can kind of see how new this space is, how much of this is actually just research.

We're starting to see new products come in. I'm really excited about your guys' solutions as you guys enter this space, the ways and the needs that you need routing for. Even on a DGX Spark, when you're doing local inference, you have more compute.

And if the memory is filled, or if the memory utilization is high, one thing you need to do is increase the compute utilization. And so one way you can do that is by spawning multiple agents that are working collaboratively.

So that collaborative piece is something that not only is optimal for all of these cloud workloads that you guys are doing, but specifically, that is how you extract more performance out of this edge hardware. And I think a question here is, and maybe to end on, is a router going to be something that we see as a product, or is that going to be seen as part of the plumbing here?

Are models going to get good at routing to other models because they know they need to be collaborative, or are harnesses going to know that they are working across multiple models?

I think we already see this. At Cognition, we're training our models to be able to be good collaborators. I think it's very clear that new frontier models, like Fable models and GPT-5.5, 5.6 models, are themselves naturally collaborative and better at delegation.

So I think we're already there at that point.

Interesting.

Yeah. I think that the systems are kind of becoming not muddied in some sense, but I think that ultimately we're understanding that as we step up the abstraction ladder and build more things to create this smarter blob, which obviously we should hopefully and we do understand how we're building it and why we're building it, that it's going to become a system that you look at kind of both the different components of the system.

But it's not just going to be just models. There's not going to be a thing as a really great harness that is in absence of a really great model and vice versa.

Yeah. Makes sense.

I think applications, especially built on non-deterministic systems like models, operate in a very low trust environment. So yes, most of the improvements will likely be distributed across both models and the harnesses. But I think overall,

there will have to be some form of controller trying to have some form of arbitration. Because even from the model perspective, you aren't in a perfectly visible world. You don't know the behavior of every model. So it's going to be at the orchestration level where you have these kind of things.

And this has traditionally been shown by other industries. Like when web launched, you had traffic-based routing, charts, different. But all the routing controls have been centralized over time.

Makes sense.

I think it's most likely going to be good news in the future. And I think caching is a big reason for that. I think to take the flip side of this argument, it might be that in the future we have one big model that's like, I know I am the most efficient at everything.

And I'm way more efficient than Haiku. I'll solve every task better than Haiku can at a lower price. Why should I ever delegate to Haiku? Something like that actually could be a model that we have in the future.

But you're always going to have these, for example, caching. It could be that you tell the model that this other model does have theright context in cache, and the orchestrator model just always has more context. And the models have to be aligned.

So I think I don't really see a world where we wouldn't be able to get models to collaborate really well. And I think they're going to get better over time, in part because they just have limited memory. So I think that's kind of one deciding factor.

And another is that there will continue to be if you just look at the rankings on OpenRouter, if you look at our public data and you look at the top model being used by dollar spent on classification tasks, guess what it is.

It's Opus. I think there are big opportunities for using small models for in-distribution easy tasks. And as time goes on, that's going to be a larger and larger percentage of tasks relative to the most valuable tasks that very smart models spend most of their time on.

Totally. Well, I want to thank you guys so much. Can we all give everyone a round of applause? Thank you.