Always-on agents run production without the on-call tax — Justin Smith, Resolve AI

Hello, hello. Hey everybody, welcome to this talk that's always on agents run production without the on-call text. My name is Justin Smith, one of the founding product engineers at Resolve AI. I've been in the space for about 15 plus years, and the sort of monitoring, observability, how do you kind of operate production systems space was at Splunk for a while.

I was one of the architects on the observability suite there, spent a good 10 year at VMware. Really, really enjoy like product design and front-end architecture. How do you, how do you, how do people experience a product or a use case or something like that?

That's the stuff I like to dabble in. But I want to talk a little bit about the first wave of AI, and it's been a fun one. I think the first big wave, and I'm sure we've all experienced this, is just how we build software.

But there's some sort of net effects of that. It's a lot of bigger PRs that are coming through. We definitely see a lot of this a lot more frequently. So people are shipping code at a much, you know, faster rate from developers, and we're beginning to see maybe from even non-developers that maybe don't actually know the code or what it's doing or the sort of like operating principles behind it.

But we're getting developer productivity, and that's good,right? That's a good thing that we're all able to sort of produce more and faster. Kind of sort of what we actually found out, and this was a survey study done, is that 70% of the time from an engineer is actually not just like, is not focused just on writing code.

It's actually spent on actually running the code that is actually shipped into production, maintaining all the platforms, scaling the infrastructure, debugging all the incidents and being on call, shipping hot fixes,right? Dealing with alerts, updating all the sort of runbooks and operating procedures, restoring services, dealing with escalations, dealing with sort of like questions from other, you know, teams and things like that.

So really coding was never the big bottleneck,right? A lot of it was really around, thank you, Granola. A lot of it was really around like how do I actually run these things sort of in production.

And that's getting harder and harder and harder. AI, it's creating a lot more issues in production as, you know, AI code sort of goes through. It's not clear we have theright sort of structures in place to deal with the amount of kind of changes that are coming through.

Unlimited tokens is sort of coming to an end. The token max,right? They're starting to clamp down. Prices are going up. Companies are getting a lot more stringent on, you know, what's being used for AI. You know, we need full stack AI.

It's not just about the models anymore. It's about the context around the models and what the models can do inside of a specific domain. These become the problem areas that we need to sort of focus and tackle on.

And this is true today,right? So it's creating more sort of complexity inside of our environment. I mean, the reality is that systems have always been complex. That's why we have, you know, these big tools that can, you know, try to give us insights into these systems.

There are multiple teams. There's multiple systems that are all having to work together, and they all have their own, you know, goals that they're trying to deliver towards, but you have organizational goals. And how do you keep all of this sort of, you know, in, get rid of this.

How do you keep all of this in balance,right? How do you, you know, pull all of this stuff together in a way that actually, actually helps you and facilitates your organization? And the answer is, well, you got to use AI inside of production to deal with sort of the amount of increase of complexity that AI is kind of putting into your product or into your system.

And so that's where Resolve, you know, this was kind of our sort of hypothesis from the beginning was, you know, we're going to see an influx in, you know, issues coming out of coding, just the increase in coding velocity.

There's going to be more need for kind of AI to actually operate and run these systems. We're, you know, lucky to work with, you know, some world-class engineering teams that are solving like really difficult problems at, you know, crazy scale.

And, you know, that gives us insight into a lot of how bigger organizations are having to deal with the influx of AI, et cetera.

Resolve itself hosts a bunch of different sort of capabilities. We have a number of agents that sort of, you know, you get to kind of experience. One of them is just an on-call agent, and this is kind of where we started,right?

So for every alert that comes in, we can do a triage of that alert. We can do kind of a full root cause investigation of that alert. And, you know, this is for anybody that's had to be on call before.

You know, on call is a nightmare,right? You're often only going on call every few weeks. You maybe don't fully understand all the changes that have come in. You don't fully understand maybe all the different systems that you're having to interact with.

And so, you know, the complexity is already there. And having an AI agent that's able to come support you and pull that context together is incredibly, incredibly valuable. And so that can often grow from just sort of getting a single page into a much larger incident across many different teams across an organization.

And we have agents there to come support the much larger activity of, you know, all of this kind of cross collaboration, et cetera, keeping everybody in sync and aligned on where the incident is happening, what the impact of that is, et cetera.

And then we also focus a lot on background agents. And this is sort of covering the long tail of, you know, what happens when there's not a fire brewing or going on at any one point. There's still lots of operational work that you as an engineer or an engineering team have to do in a lot of ceremonies of, you know, passing context off or dealing with kind of one-off issues or kind of having to scratch that itch in the back of your head of like, is that part of the system okay or not okay?

And you're constantly having to sort of bounce across all these different things. Underneath all that, you know, we have an agent architecture that deals with models and context and reasoning and actions. Learning is, I'll sort of like half pause on that one.

I think, you know, some of the biggest issues that we've seen, it's not that a model by itself is not smart or whatever. I mean, models have gotten incredibly capable over the last year, let's say,right? But especially over the last like six months or so.

But the idea of understanding, like truly understanding your environment and the way that your services interact and where the hotspots are, keeping track of all of that sort of understanding is incredibly difficult, but it's incredibly important for any model to be successful at, you know, the tasks that it needs to do.

It has to have an underlying sort of learning system to be able to capture that knowledge and that sort of understanding of how your system operates. And so we spend a lot of time thinking about how do we have systems that not just can understand your environment at any one point, but grow as your system evolves.

Because again, your system is evolving faster and faster. We need to keep up with learning about what's the current state, what's the current sort of causal change that we need to be sort of keeping an eye on. And then of course, all the enterprise stuff sort of underneath.

And so this is that same view kind of packed out.

Come on. Okay. But so today, so we do a lot at Resolve, the on-call and the incident stuff. I'm going to focus a lot more just on the background agent stuff. So how do we deal with the things that maybe aren't sort of immediate fires?

If you have questions about the immediate fire stuff, we have a booth down in the expo. Please come check it out. Our team would love to demo to you, et cetera. But today we're going to focus on the background agent.

So kind of a little pop quiz. Feel free to raise your hands. Is anybody using agents as part of your daily workflow, maybe outside of the coding? I'm assuming everybody's doing coding agents these days. Is anybody doing like actually running sort of agents that are sort of helping in other ways?

Okay. Decent on it. Any good examples? Any fun stuff that anybody has? You can just yell it out.

Meeting review.

Meeting reviews? Yeah, meeting reviews. I just had my granola show up and.

Market research.

Market research. I do a lot of, let's have a good conversation about it. Yeah, yeah, I do that all the time. Any other ones? Maybe one more.

Therapy.

What's the one?

Therapy.

I can't hear it.

Therapy.

Therapy. That's a fantastic one, actually. We are humans here today. This is very important. That's actually, that's a very good one. Okay. So people are having some stuff going on. So, you know, and this kind of recaps a little bit again.

A lot of production work is not about, there's not a sort of big ceremony that everyone is focused on for the type of work that we have to do. On call, you've got a page that goes off. You know somebody's going to receive that.

Incidents, you create a bridge, you invite people in. That's great. But there's just a long tail of other things that we are accountable for that doesn't have sort of a thing that's going to show up in your sort of job description of like, this is what you're going to be, you know, responsible for.

Watching deploys that go out and make sure that they're actually getting out healthy. A morning report or incident digest of just like, what's the state of my system today so that we're all on the same page? Hey, that P99 drift kind of came back.

Is somebody looking at that or not? And, you know, this is pulling people in to try to like figure out what's going on. This may not be paging,right? Because we don't, we're not going to alert on everything. Produce the capacity report,right?

Like, are we tracking okay,right? This is maybe a company goal this quarter. Are we tracking against that? Somebody's going to have to be responsible for doing that. The recurring health check and just kind of checking and making sure things are kind of running okay and not waiting for a customer to complain first.

So this work doesn't have like an obvious like, oh, this, you know, this now needs to go be done. But it's work that we end up having to do. So what is a task? A task is just execution and the context to understand how to actually execute the task.

Execution is very, very important. It's understanding what to do and being able to execute that. Maybe having access to the tools, et cetera,right? Obviously very important to do. But we think the production context is just way more important because it's one thing to go check a dashboard.

It's another thing to say that metric smells off. And the execution can load the dashboard. It's the production context that's going to say, this feels wrong. And I don't know if I can even explain why it feels wrong.

It just feels wrong. And I want to dig into the next layer of sort of understanding of that. And so really, if we start talking about background agents and being able to kind of perform tasks, you need both of these.

You need the execution engine. That's great. But you really need that production context that tells you, is this important or not important?

So every background agent, you know, there's a few different principles that we like to think about with our background agents. When does it work? How does it work? How does it know what to go do? When does the agent work?

It can work in a bunch of different ways. It can just do it on a schedule. Maybe this is the morning report, et cetera. Just kind of do some summarization for me kind of on an ongoing basis. Maybe it's a weekly event,right?

We do an on-call handover every Thursday. And so a lot of the work that our agent does is sort of prepare. Like what are the kind of interesting trends from the last week that the next on caller needs to sort of understand as they pick up the rotation?

Event streams. So, you know, there's lots of systems that will sort of push events as kind of key things happen. So deployments go through a CI/CD pipeline. There's other sort of Slack-based,right? We get a lot of Slack things, messages coming through, et cetera.

And these are things that we can sort of pick up and trigger and say, oh, if this event happens, let me sort of understand what that event is and go do some work. And then message-based. So I could just tell it, hey, go do some work and it will go do some work.

That's fantastic. How does it run? Always runs. It's in the cloud. So if you close your laptop, it's okay. Runs inside of a sandbox. It has kind of a file system underneath it. This allows it to sort of self-organize a lot of its work, et cetera, as it's doing things.

And then obviously back to the learning loop,right? So that idea of knowledge and sort of a memory system underneath that to really understand your systems and as it's doing a task, able to sort of reflect on that task and, you know, do a better job next time or the things that it learned from one task, it can sort of apply into a different task because again, this sort of shared sort of knowledge system works across all the different tasks that we have.

So how does the agent know what to do? It has a task system. It can pull in all the skills that you have in other systems. That's fine. You can connect those. And it's got obviously the integrations that it's going to plug into.

So let's talk a little bit about what types of things you can hand over. We've got four sort of workloads that we're going to talk about. But if you think about the previous couple of slides, these are sort of very basic primitives that we've built into the system.

You can get very creative. We have a number of people inside of Resolve that have gotten very creative with the type of sort of background activities that things that they have. So I want you to use these as kind of, these are things we've seen be very successful inside of Resolve, but also with, you know, a number of our customers.

But, you know, sky's the limit and you can get really creative. So deployment monitoring. So this is a big one. You know, any change inside of your environment is an opportunity for something to go wonky. And so, you know, having an agent that is able to watch as all these change events come in just to do a sanity check of is everything stable is incredibly, incredibly important.

And, you know, a lot of people have a decent CI/CD system. I mean, this is like tried and true stuff that we've had as an industry for quite a while. But we notice a couple of gaps, you know, from in most of our customers.

You know, typically the checks that it does are good. They're good baselines, but it's not exhaustive based on the type of changes that are going in, et cetera. There's certain signals you'd want to watch or not want to watch.

And so every rollout is a bit unique. Oftentimes you have change systems that you're not piping through a CI/CD system, like a feature flag or maybe some infra changes that might happen, which maybe don't get any monitoring at all.

And you're sort of just trusting that an alert might fire and an on caller will wake up and say, who changed what,right? And so deployment monitoring is actually a really big use case that we suggest people sort of go through.

And I'll show some examples of that in a second. Schedule health and anomaly checks. So this is just sort of the ongoing periodic checking of some of your systems. And, you know, this is maybe something where it's like, go check, you know, sort of my general dashboards on a routine basis, maybe every morning, just kind of do a casual check just to make sure there's nothing kind of weird from last night that I might need to be aware of.

But this can also just be sort of a time-based thing. Like I made a change in part of our system. I'm worried about this, you know, third-party service that I'm kind of interacting with. Let me just kind of set an agent to kind of watch that maybe for the next week just to make sure everything is kind of stable.

And then that agent can sort of, you know, stop its job. Operational reports and handoffs. I talked a little bit about this. These are the sort of ceremonial things that we might want to do just to, you know, spread information, summarize things, kind of bring things to the ford.

And then a first responder to engineering questions. And this one's kind of fun a little bit because the trigger for this is actually just a Slack message. And I will say one of my biggest, like let's call it responsibilities as an engineer is watching all my Slack channels and trying to make sure everyone's kind of happy and like that nobody has any burning questions or anything like that.

And so I can be sort of heads down trying to build something. And then, you know, the, you know, eventual sort of Slack notification comes in that like this channel, somebody asked this sort of kind of important question.

And I just need to jump in there and try to provide context, et cetera. It's not hard work. It's not hard for me to go answer questions, but it's disrupting me. And if I don't go answer it, they won't get an answer for a while.

And what we found is like our agent actually has access to a lot of information that people ask questions about. At least this is true internally. And so we actually have an agent that can watch all of these sort of critical channels and determine whether it has enough sort of confidence to answer the question or not.

And one of the fun things is like our agents have access to like Slack DMs and things like that. And so you can have an agent that basically will DM you to say, I think I know the answer to this, but I'm not sure.

Can you confirm this for me before I, you know, respond back? So this kind of emergent behavior gets kind of fun and interesting as you kind of build these things out. Okay. So I'm going to flip over and hope all of this works.

Cool.

So let me see if I can find the one that I wanted to show. So this is our sort of demo application running in our sort of demo sort of Slack environment. And what I wanted to show off was some of our deployment stuff and talk a little bit more about what's kind of going on under the hood.

So this is, you know, sort of a fake environment just to kind of showcase some things. So here, anytime somebody posts a sort of GitHub tag, our agent's going to sort of see that and say, oh, I should, that's a release.

Is that a release? Yes, that is a release. Let me go watch that. But it's not just going to watch it. It's going to do something that's slightly more intelligent. Because like I said before, everybody kind of has a CI/CD system.

It will do the sort of standard checks on, you know, certain KPIs. But what the agent is able to do is actually look at the changes that are going in, understand what telemetry might help us evaluate whether those changes are, you know, good or not good or like are putting the system in an abnormal state and build a sort of customized plan that it's going to check just for this specific release.

And this is why I kind of go back to like, you know, our goal is not to sit here and say, we're going to replace an entire CI/CD pipeline. You've spent time organizing that. But this can sort of patch a lot of, you know, parts of your system that may not be as robust as they should be.

And it would be great if you had a single engineer just focused on like watching all the things on every release, but that's really expensive. There's a lot of cognitive load. You'd rather have them doing other things. So now the agent can come and actually do a lot of that sort of dynamic understanding of this is the change.

So I'm going to sort of check for these things. And so here, the checkout replaces currency service. You know, we'll monitor the checkout latency and the error rates. Let's take a look at the Kafka pipeline because that's sort of involved.

This is the sort of causal chain I want to sort of say I want to make sure is healthy. And it'll check that. And it can check it not just once, but sort of on an ongoing basis. And, you know, none of this is hard-coded in.

It's not like, oh, let's just wait for 15 minutes and then try this again and then we'll be done. Again, the agent has a bit more autonomy and you get to guide it a bit on how much autonomy you want it to have.

But it could decide I want to wait for another hour because this type of issue might only hit every, you know, every so often. So I really want to spend a little bit more time focused on this. Maybe I'll come back in three days and say, is this deploy still kind of healthy?

Are we seeing the change and the effect that I expected to see out of this? So this is the type of thing that we can bring. And this again works for feature flags, for interchanges, sort of any sort of eventing system that you can think of.

The sort of on-call handoff reports. Let me scroll up just a little bit. So this is just a summarization of all of the work that was done, you know, over the last day that the agent is kind of just summarizing up.

And this one's a little bit verbose, but you can see a bunch of different sort of investigation summaries that we did, some notable changes, et cetera, work completed, et cetera, critical open. You know, I guess that's the on-call handoff.

But one of the nice things, and I don't know if I'll be able to watch it go all the way, but you can always just come back in this thread and, you know, this is too verbose. Make it shorter.

And I'm not going to be able, unfortunately, no time to watch this actually go, but this works. The agent is able to update its task underneath and able to sort of give you the answer, like update so that the next time it fires, it's not going to be as verbose.

And I can tell it explicitly what I want, et cetera. I was just kind of giving you an example. This is kind of the more fun one. So, you know, here I'm just like posting different problems. I'm not having to know that Resolve exists.

I don't have to like mention Resolve, whatever. I've set up this agent to sort of passively watch this channel. If you see something that you think you have an answer for that somebody's kind of, you know, digging into, go ahead and respond.

Otherwise, don't. So, you know, here's a message that I posted that it's decided I don't need to respond to this. So again, very kind of flexible system that can kind of adapt to a bunch of different things.

Uh-oh, where'd my slides go?

We have in the UI, there's a bunch of stuff that you can do. You know, you can always go down and inspect all the different tasks that you have and view their reports, view previous runs. You can see all the work that the agent has done underneath to sort of accomplish that task.

So you get a lot of visibility into what the agent is doing, but we think the surface area being where you live,right? So Slack is this like kind of, or MS Teams if you're on MS Teams, as this kind of first-party experience to sort of integrate the agent into is incredibly important.

And so, you know, how would you get this stuff sort of set up? It's really just through talking with the agent. And so here, this is me sort of saying, hey, I want to do a new recurring health summary for my team.

So the agent's going to take a look at my environment. It's going to explore my environment a little bit and eventually likely come back and ask me a couple of questions about what I want to see, what kind of report do I want, how verbose do I want it, et cetera.

The agent's going to go ahead and do all that. And it's going to set up that sort of initial thing for me so that I can test it out, make sure it's working, and then share it with the rest of my team.

In the interest of time, I don't think we'll get to this, but come by the booth and you can see more.

Cool.

So that's background agents. And again, I sort of lean back on be creative,right? Like everyone has unique work. I mean, as a company, we believe that every, you know, every company is a unique place. That's why we spend so much time on our knowledge system, et cetera.

Truly understanding your, you know, what your environment looks like, what your needs are, et cetera. That begins to kind of tell you where the biggest benefit from having these agents begin to pick up work would be. Also just to call out, if you have an agent harness, if you, you know, internally, if you're building your own, everything that I showed is accessible through kind of MCP servers, et cetera.

So you can really graft Resolve into kind of any system that you have as just kind of an extension of learning to kind of do deeper work or to sort of pull production context a bit more efficiently or, you know, even augmented with all that learning stuff that we've done.

And then obviously bring your own skills along for the ride. Don't go duplicate a bunch of stuff. So really the biggest things to take away, cost operational work, it's not navigating, you know, it's not just in the task execution, it's in the environment complexity,right?

That's where the biggest issue is going to happen. Background agents, they run on schedules, they run on triggers, they're very composable. You can sort of graft them into lots of different use cases. It's fun to see people explore that.

And yeah, you open Resolve just to kind of see what the top findings are, et cetera. But, you know, ideally a lot of your interaction is kind of in the places that you're already kind of doing work. So if you have any questions, you can find me down at the booth or, you know, just meet me out in the hall.

But thanks for coming. Appreciate it.