Agents Are Where Microservices Were in 2015 — Roberto Milev & Uday Kanagala, Navan

Right. Hello, everybody. Um, welcome to our talk. My name is Roberto Milev, I'm the chief architect at Navan, and I have Uday here, who's also part of the architecture team. Uh, Navan is a travel and expense management company, and we'll share with you some of our learnings around how we run an AI and what have we, uh, discovered.

So, if you've been long enough, uh, in this industry, you remember that over time there are a few paradigm shifts, and we all tend to jump on a bandwagon and try to, uh, kind of do thingsright. Last time was when we all jumped on the microservices bandwagon, and out of that, a lot of good things came out, like container orchestration, Kubernetes.

Then we had service mesh, uh, uh, circuit breakers, all of those good things. But it didn't happen overnight. Like, it took a long time. It took some time for us to learn how to do these things. So one of the quotes from there is, "If you can't build a well-structured monolith, why even try to build microservices?"

Uh, it kind of translates today because if you can't build a single agentic loop, why go in and try to build a multi-agent orchestrated, uh, system? So over time, just like previously, uh, a reference architecture is emerging. So we have learned a few things by, by doing in production.

We have a lot of agents, a lot of tokens per day being used. And as I said, there are a few layers that have standardized, that have crystallized around what do we need to run agentic flows reliably in production: Runtime, Memory, Context Management, all around o-operational cross-cutting concerns, and around orchestration as well.

So today we'll go over some of these layers, all of these layers actually, and we will show, uh, kind of where the industry is, what we have done, what we have learned, uh, and, and so on. So starting at the Runtime layer, uh, we've talked a lot and we've built a lot of services in order to scale them, uh, statelessly before.

And now we're in a new world where, you know, agents are stateful by nature. They need to have, uh, persistent, uh, sessions. They need to have isolation. Their lifecycle is different than the lifecycle of a traditional API service, and so on.

So, uh, the cloud providers have jumped in and tried to fill this gap. Um, you know, uh, AWS, uh, GCP, Azure, they all have a some incarnation of, uh, agentic runtime. If you scan the QR code for this slide and for the following slides, you will see a comparison of some of the features and how different cloud providers try to, try to, uh, approach this.

At Navan, we run everything on AWS. Uh, AWS has an agent core runtime. We heavily use that, but we have filled some gaps around that. Like the, uh, session persistence and rehydration is something that we have built. And we also run a bunch of other, uh, a bunch of, uh, SDKs for writing agents.

And part of, uh, these runtimes is typically they are framework agnostic, a-although they all prefer their native, uh, framework in a way.

Uh, the next, uh, layer in the stack is around, uh, Memory. Um, we started with RAG. RAG was kind of a big thing for a while. Uh, we were kind of driven to that out of necessity because you cannot fit an unlimited amount of context into an agent.

And over time, uh, all of these cloud providers and the industry has implemented a pipeline where memory is kind of automatically generated by following a workflow of ingestion, extraction, and then consolidation and retrieval. And there are parts of RAG that are built in things like a long-term memory that inherently has some semantic characteristics, but memory is built up over time, from short-term conversational memory to long-term memory that you kind of manage yourself.

Uh, then episodic memories about, uh, kind of instances that worked well and didn't work well, uh, and so on. We at Navan, again, being an AWS shop, um, utilize their agent core memory, but we are also kind of doing it in a way that, uh, matches, matches our, our use case.

And then the next thing is context management. You know, it's a hot topic. It was a hot topic, and it's still a hot topic. Context windows are growing bigger, but there's never enough context. Or if there is too much context, again, agents struggle with that 'cause you lose focus and so on.

Um, what we found working is that, uh, focusing on skills as a unit of context. And I'll explain what I mean by that. Uh, we look at skills as both having context, meaning instructions and, uh, setup about a certain domain or a task.

And there's also the, uh, the second part of the skill, which is the tool execution and, you know, the agentic part. And we compose context dynamically out of skills that we, uh, use as units of work that are, uh, pluggable, that we can test independently, and that we can reuse.

So, for example, when we are we have an agent, we have skills that are, uh, that are specific to a domain, and based off that, we compose them. And we rely on the, uh, uh, you know, the progressive disclosure, which is a feature of the skills itself, to start with a limited scope of context and then, uh, expand by including, uh, metadata, uh, further down the, the line.

Uh, I'll hand it over to Uday, Uday now to kind of walk us through the rest of this.

Thanks, Rudo. Allright. Uh, can I have a quick show of hands, um, here? Who have, who have built an agent, uh, which failed halfway through, uh, multi, uh, 20, 20-step or 30-step process, and be able to figure out quickly, uh, or reason about why the agent failed?

So again, logs. We've generally been traditionally with microservices. We all are familiar with logs. There's logs out there, and then we go check out the logs. But this changes everything the moment we switch to agents. Agents output a lot of thinking.

There's too much to consume. So that's not theright way to do it,right? So traditionally, that was the way, but our thought has to be changedright now in the, in the way the Claude, as an example, when we take Claude as an example for an agent, there's hooks, and we can intercept everything that Claude, as an agent, does at that level.

So what kind of tool it calls,right? What kind of decision it's making. So before pre-tool and post-tool call, or a pre-session or a post-session. So all of that are a point, point in time for us to intercept and make a decision and either block, do a, uh, blocking operation, or to log a, uh, metric or emit a metric,right?

So this is a critical, uh, place where we can emit or traces. At Navan, we use, uh, one of our, uh, provider, BrainTrust, to emit these, uh, or traces. And through these traces, we should be able to figure out the spans, the traces, and at what point in time where the agent is stuck, which gives much more, uh, confidence into, um, how we operate and build the agents.

This is a day-to-day operational challenge. Um, building agent these days, there's so many frameworks, but how do you navigate building and operating an agent later is the primary concern now. And moreover, uh, the reasoning chain, uh, the thought process, and, uh, the critical signals that we emit, uh, here.

Uh, as part of the trace, uh, captures, we emit a few, uh, primary signals, uh, here. What is the current goal the agent is going through, uh, the reasons behind its operations, and the belief status, and, uh, the tool calls that it's making.

So this kind of gives us our judgment pointers, um, uh, in the traces. And when we make, uh, when the agent makes, uh, a decision, um, there is a confidence score, how confident it is when it makes this judgment,right?

So whether there are multiple paths that it leads to this, uh, choice, or whether this is an inferred answer. So basically, these are signals that give us, uh, confidence later to review. If this is an inferred answer, there could be a human in the loop to guide through, uh, and tweak the agent to, uh, perform a little better.

Again, um, can I have a raise of hands again to see, uh, how confident are you, like, 100% confident in, uh, testing pipelines with your agents? Right? So this is one of the other, um, critical aspects, uh, today, um,

because agents are nondeterministic. We've all been used to program and write much more deterministic flows. And we know how it works. The when I ask an engineer, an engineer can come and tell me how this the algorithm, the sequence of operations, everything is programmed in our mind.

Everything is expectations. But now the agents come into a nondeterministic way, and how do we test them,right? So that is very criticality here. And yeah, we are also struggling. Uh, we've, uh, started doing building agents. We the day two operations was challenging, and then we failed, uh, in a lot of steps.

How do we course correct? The moment we change something, something else broke, uh, breaks,right? So how do we do that? Um, one, one approach that we took, uh, this is from, uh, research papers, uh, around the traj concept of trajectory evals.

Like, in a multi-step, uh, orchestration, when an agent makes, uh, 30 steps or a decisions to make to reach to a goal, if that is a program or that's a different story. But this is not a program. This is nondeterministic way of it makes up its own, uh, steps every time, uh, differently.

So how can we chart a deterministic graph here? Is it possible? No. Can we have a trajectory of its starting from an end to a goal, and then see how much, how far it went in the trajectory, and how far it went from the source to the destination is what we can compute to evaluate the efficiency or the completeness of the, um, agent, uh, agent evaluation.

So we, we heavily rely on, um, trajectory evals. Um, and, um, this there are few other signals, uh, as I briefly spoke around, uh, in the previous slide around the inferred signal. Um, if the answer is from an in-in- inferred answer, uh, how can we, um, loop that into, um, and make a signals around, uh, how can we classify that this is a regression and make fixes towards the agent?

Right. Uh,

so the next is the, um, guardrails. Um, where

is this the one? Yeah. So gu-guardrails and authorization. Um, this is, uh, critical. This plays a critical role in enterprise AI. A lot of information is being piped to models. Um, there could be sensitive information that goes into it, uh, without our knowledge.

And we as, uh, uh, leaders, how can we put in this governance layer, um, to stop this, um, is very, uh, cr-critical here. And, and the concept of, uh, authentication and authorization, um, is taking up a different approach here.

Um, traditionally, we've seen, um, a user or a service account, but now what is an agent? Agent can be acting as on behalf of users. There is so much of things, uh, so many of use cases there. Hey, book me a flight whenever it's cheaper than $200,right?

So we just tell this assertion, and then agent go figures out and does this action on behalf of me. So is it me making this purchase, or is it agent me making on behalf of me? So there is agent acts as a on behalf of user, or agent use a service account as well.

So the line is being blurred here, and we need to make fine-grained authorization decisions here. And the policy layer, um, that's where the guardrails and authentication authorization plays a critical role. Um, and in Navan, uh, what we employ here is, um, before every tool call, pre-tool and post-tool, we have these guardrails, uh, to check and block, uh, and make, uh, informed decisions.

And this, uh, single agent versus multi-agent, again, this is kind of, uh, orchestration wars you can think of, whether to build a single agent or a multi-agent. Um, again, as Roberto briefly hinted, if you can't perfect and build, um, single agent, why go towards, uh, multi-agent,right?

So learn from our, uh, failures, experiences, and build towards that. At Navan, um, yeah, what the approach that we have taken is, uh, single master, uh, and then we adopted, uh, subskills. Um, there are sub-agents within it. Um, so it's a single agent that can, um, progressively load the skills and understand decisively what needs to be loaded into the context and then, um, make this, uh, navigation, um, through the use case.

But there are other patterns that are also emerging. There are different class of use cases here. One is, um, agent-to-agent communication. So there are if you take a large-scale organization, and there are so many of these teams that are, uh, that are acting as a boundaries, and they don't talk to each other, let's say, how do we communicate?

There are two agents on either of this side,right? How do we do it? So there is a gateway protocol which can, uh, help us establish the contracts in terms of skills. And we can use A to A as a protocol there, uh, which kind of, uh, is a boundary between the teams.

Yeah. Over to you, Rudo.

Allright. So as we went through the stack, it's obvious that, um, some components of the stack are in a more mature state, and we already have good answers for them. As I said, the runtime, I think it's pretty much solved.

We are so advanced in orchestration, and we are running LLMs in kind of, uh, a very, uh, brute force way. So scaling is not a not a problem. Also, memory, I think, uh, as, uh, the frontier LLMs get better and as our practices get better, we will, uh, find a way to cover the majority of the use cases, and there is good maturity around the, the cloud providers.

Uh, MCP has emerged as a de facto protocol, and tool calling is now a feature that everybody supports. So we are seeing some industry convergence, uh, around, uh, that as well. And MCP as a standard is also evolving.

Now it's becoming stateless. It's, uh, we are reaching a point where kind of we know how to invoke services and, and, and tools, uh, with agents. Uh, in some areas, things are happening, but, you know, there's still a lot of unknown.

Around observability, there is a push towards OTel, but does OTel really work for agentic calls? Uh, yeah, you can make it work, as Uday was saying. Um, also, we are getting more comfortable around, um, around the, the, the testing patterns.

It's very hard to test, but we have found a way to give customers, um, quality experiences, even with the unreliability of agentic system. And I think that's kind of, uh, getting in a in a state that is, uh, that is more better defined.

Orchestration is another one, um, where, you know, we have a pa we have patterns. Uh, we can build, you know, bigger agents, smaller agents. Uh, as we said previously, probably theright answer is to not over-engineer. Uh, so we are learning there, and, and, and, uh, a pattern as core thought is also emerging.

Uh, where we're all struggling with, and the previous talk was about this for the developer, um, AI-assisted development perspective, but also we are seeing these, uh, issues from our production agents. It's very hard to predict cost, and it's very hard to manage cost, uh, and put guardrails and solve this in a way where there is a reliable maybe fallback or have agents be, uh, using cheaper models for certain tasks.

Uh, uh, this is all driven by kind of the big AI vendors who I think their interest is for us all to spend more tokens. Um, replay and debugging, Uday talked about that. That's also a big, big issue.

It's very hard to understand, but I think this is also something that, that is gonna be solved because we can now use, uh, agents to, uh, get over the cognitive overload of trying to debug what they do. And then standards.

Um, standards are emerging, uh, by, you know, the community. Uh, OTel, as I mentioned, agent-to-agent is young. It's kind of pushed by certain vendors, but I think over time we will we will get, uh, there. Uh, with all of this said, you know, we know what we need, and it's up to us to go ahead and build it.

Thank you, everybody.