Agent Frameworks Considered Harmful — Rémi Louf, .txt

Hi everyone. So, originally I thought I was going to give a very technical talk, but I saw it was in the leadership track, which I'm not sure what it means, but I was like, okay, I'm going to do half high level and half technical.

So it's more a story about what I, you know, what I did in January, because in around December, um, agents kind of became really good. You know, there was a step function, something happened, I think it was Opus 4.6, and that's when I realized, and I work in AI, where I was like, okay, this thing is really happening.

And so I took two weeks out, I took two weeks away, so I'm the CEO of .txt, you know, which is a 15 people company. I just told my CTO, I was like, okay, I'm just going to go away for two weeks and I'm just going to dive in this thing and try to understand what we can get out of it and how good it is.

And so the story is, you know, it is the story of me scratching my own etch for two weeks and trying to figure out how we can use, actually use agents, and what are good primitives to build agents, and whether, you know, it already exists.

This was a really clean bait title, but actually it turns out to be a good title even for this talk. So what you can see here on the left, the castle, is my office. That's true. I do rent an office in that castle.

And the small thing with an arrow that you can see is like this robot mower, which kind of works unattended all day, every day. It just does its stuff in the background without anyone having to use a remote control or think about it or anything.

And I kind of wanted the same thing for my morning, because my mornings are always the same thing. The first couple hours it's browse, market news, review of like Linear, could be Jira, my CRM. And also I like to walk for about an hour in the morning and then the next hour is spent trying to process the really long voice note that, you know, was recorded while walking.

And you know,

all I wanted was my morning briefing with my coffee. And that's kind of what we've been told for a couple of years, like what the future would be. But then when you really start working with it, even if you're not coding, all you get is a TUI today.

So it's amazing. You can code, you can actually, you know, I started doing things that were not coding and they're great for this, agents are great for this. But it's kind of the equivalent of having a robot, like a tractor mower, that you still have to stay on even if it's driving by itself,right?

It's kind of very frustrating because you have to, it can do many things, but you still have to be on. And so, of course, the labs didn't stop there and they came up with apps, which I call basically SSH with vibes.

That's great, but in this situation, when that came up, I was like, wow, that's awesome. I don't have to use like a term, like SSH on my phone anymore. Codex is great. However, I noticed I just started, you know, I was on my walk and I was just instructing the agent to do things while I was walking and so I wasn't thinking, you know, very clearly anymore.

I just started running agents on my phone during my morning walk. And this is not great because this is the equivalent of this, is you're kind of midway. It's not the tractor that you have to stay on. It can actually do something without you beingright next to it, but you still have this remote control that, you know, you kind of have to change your trajectory every now and then.

That's useful. It's kind of absurd when you think about it. And actually when you look at people, like on their phone all the time, just doing this, it's kind of absurd and it's clearly transitional. Like surely we're not, it's not, it's not going to stop there.

And so I did a very dumb thing as a CEO, which is I started coding. Don't tell my board. And I started to build the dumbest thing that could possibly work. And of course it became a really crazy rabbit hole.

The repo is there if you want to take a look at it. The code is not amazing, but it works. So the first thing is that, you know, I started using frameworks. I mean, there are great frameworks and I'm going to name any frameworks because they're all good in their own way and they all have flows in their own way, which is fine.

But I spent all my time actually editing the prompt within the code and I was like, this is actually not very useful. So I'm like everyone here, I hate YAML, like the next guy, but I still found that this was actually a lot easier to start implementing agents without code.

You can version it, you can diff it, you can review in the PR, but it's just, and it's just so easy. You can just, you know, write your file, you drop it in a folder and then it just magically appears once you have the runtime and it just magically works.

And, you know, then I needed like my market watch to run every morning while I'm, you know, while I'm walking in the field. And for that, we have things that, you know, have been around for a while, which is cron jobs.

And schedules specify, you know, when the agents need to be run. And also we'll see, it's very important later, they publish, they publish events. And, you know, markdown and cron, obviously, you know, it's much more complicated than that under the hood, but the interface is this.

You don't write code. And that's the whole product so far. And honestly, it just mostly worked at this point. I'll come back on mostly later. And so this is actually a real picture of my, one of my morning walks.

And so what I do is I record voice notes while I'm walking. But cron, you know, cron jobs, I mean, people would use cron jobs for this because that's what's available in Codex today, but they're not ideal because they cover when, but this is just one point in time.

It doesn't cover because this happened. And, you know, things that happened in our system, like automatically when you drop the voice note now in the system, it will emit an event and an agent will react to that event.

And it's the same thing when you have a new email, a new entry in the CRM, I mean, anything, a new PR that's open, a new PR that's merged, etc. It just reacts to events. It's not just a cron job.

And that means that, you know, agents are the voice note processor. It's just, you know, not a cron job, but here you have accepts and returns. So it just declare what it accepts and what it returns as an event.

And here it accepts a voice note, transcribes it, turns it into durable notes on theright, and it emits a new event. And for that, I use a structured output. So we'll come back to this. And, you know, now we finally have the future we're promised because that voice note agent emits a voice note processed.

And then I have my daily brief agent that actually will take the output of the cron job, will take the output of the voice note agents, and we create my daily brief, which is posted as a Slack message.

So the Slack message.post event is actually, is actually like a process actually subscribes to this and emits and sends a Slack message to me. It's actually, this is a real, this is a real thing. It's working. I can show you after on my phone.

And, you know, there are frameworks that are going to tell you the fact that you need graphs for this and code. You do not need graph in this case. All you need is events. You have no edges to maintain.

Agents simply subscribe to events. Anyone can come in and edit this. You don't need to, yeah, you don't need to know how to code. You just need to know what events exist in the system. Finding and find out are free.

No code and it's just drop a file and the topology emerges, whatever the log says happened. And, you know, then of course I tried to run it. So the first version took about, I mean, you know, I cheated, I cheated.

I used, I used Codex and it took about like a day to write like the first thing out of my week. But of course I tried it and it broke. So these are real examples actually. The dates know, but it's real examples.

It's like the first day, daily brief was posted to Slack twice. On Wednesday, one of my voice notes completely vanished. And then, you know, towards the end of the week, I kind of like played with the prompts all week and the market brief was garbage, but I didn't version, I didn't version my changes and I couldn't remember actually what I changed in the prompt that made the thing completely useless.

Now, if there are distributed or ex-distributed engineers in the room, you probably know this shopping list already. There is nothing new under the sun. And, you know, each failure, so each of these failure modes that you found actually led to building one piece of what turned out to be a runtime.

So the last note actually turned into a log. I just wanted everything to be saved forever so that I could go back to it and look into, into what happened. The duplicates, it was because I was not following, you know, it did several attempts and I was not following them.

I didn't have a proper queue. I wasn't counting the attempts, etc. And then probably the most interesting part is the last prompt, I got into a really deep rabbit hole in there and I just ended up building a content, like a content addressed system for this.

Content addressed system, you can think of Git, you can think of Nix and any other build system. And, you know, that was, and I didn't do this because I wanted to design a runtime. I mean, by that point, I still just wanted my agents to work.

And I also liked the distraction. And I just paid off debt as it appeared, like errors as they appeared. I hope my board won't see this talk. So the log, the log is the system's memory. Nothing is lost and everything is observed.

You can, you know, you only have one append only event stable. On the left, it's a real common line, like command, zeta events, and you get all the events. They are causally linked as well. Like, you know, which event triggered which event, which happens to be super useful when you're debugging.

And, you know, even with three, four agents, you start having like major debugging headaches. So that was super, super helpful.

And everything is queryable, which again, for debugging. The second thing is, you know, okay, we have a log, so we can trace back things, etc. But it's still really hard to know what went into the, like what went to the model, what prompt was sent to the model again.

Because what you see when you're using Codex, it's kind of a lie. Like you kind of have like a live chat session with the model. And so you tend to think that, oh, that's what the model saw. And, you know, that's exactly, so I can understand what happened.

The truth is, that's not exactly what the model saw. There are many reasons for that. One is, I mean, compaction obviously is a big thing, but also, you know, there are just quirks. Also, you know, OpenAI doesn't share, or Anthropic for that matter, don't share the thinking with you, the thinking traces.

So you have no idea. I mean, kind of have an idea of what went in, but not completely either. And so you need something different. You need something different. And that was the big rabbit hole, which is trying to find a way to build a system where you can trace back to what the model saw internally.

And so what I did was basically built, I mean, nothing new. This is basically how a build system works. So you have different parts for a prompt. You have your system prompt. You have a description of your first skill, of a second skill.

Then you have the description of your tools. You have your user message, which is the question to the model. Each one of those is stored and addressed and, you know, stored somewhere as a identifier, which is a hash.

And so when we build a prompt, instead of building a piece of, I mean, before rendering the text, we actually represent the prompt as a list of these, of these hashes. And so what that means is that down the line, when I have a model answer, which by the way is also stored in the same way, we can trace back to the prompt very easily.

And then from that prompt, we can know exactly what went into the model's context, which actually matters a lot. I mean, it matters a lot for debugging, but it also matters, I mean, it makes compaction a lot easier.

You're just manipulating a graph,right? You're not manipulating strings. It's just a lot easier. And it makes KV cache management a lot easier as well indirectly. And, but I think that when, you know, I guess probably

the main advantage that's when you use that skill is really auditability. It's like you can know exactly what happened with that agent and why it returned what it returned. And so, you know, I'm just going to go pretty, pretty quickly over this.

What you get once you have this graph is you get diffs. Like you can say, okay, what changed between these two runs? Like which, which components changed? Was it just my message? Did I like give the model a different skill?

Did I give it a different tool? So you can just, yeah, you can just run this function and it will show you, you know, the difference between the runs. So here you have, you know, three components that were identical.

There's one which is, you know, the user message changed. And then you had all these other messages that were actually, you know, that were continuing, it's continuation of a single session. Then you have another thing for free, which is replace.

Replace turned out to be really useful for me because after a while, I mean, when I saw the cost ramp up, like the thing when you have observability is that you do realize that cost increased very quickly. I wanted to try with open source models.

And so I wanted to rebuild all requests to eval and see if I got the same thing out, if I got something satisfactory, if I needed to change anything. And turns out that once you have, you know, this content addressing system, you can rebuild the request from the graph and you can just replay it exactly the same.

And you can, you know, resend, you can use a different model. You can use a different request if you want. You can, you can change it. And so, yeah, you get actually a lot of things, I mean, for free.

You need to implement the thing. And so this is kind of different from what you find, I mean, what I found when I started doing this. It might be different today because it was a couple of months ago.

Is that out there you had a lot of libraries. So it's just frameworks and frameworks just call code. Your agents leave inside their abstractions. And I don't like analogies with, you know, operating system. Okay, everyone has used that analogy, but okay, let's say a kernel, like runs processes and your agent kind of is a process.

It doesn't matter what it does actually, but the system can schedule it, can build to isolate it, it can isolate it and journals it with the log. And the agent definition, so the markdown is user land. Like you don't need to use it with that system if you don't want to.

Actually, you have a front end that doesn't use this markdown, this markdown format at all. And okay, here's a very important point. And, you know, that's kind of a takeaway. And it's also what justifies me working on this because disclaimer structured outputs is our specialty.

We've been working on this for three years. And it just ended up being a big dog fooding project. And the reason why I did this at the beginning was not because I absolutely wanted to use our software. I didn't necessarily want to, you know, fork Llama.cpp to other software, etc.

It's just because Anthropic was terrible at structured outputs. And so like 20% of my events were wrong and were rejected by the system. So that's why I ended up doing this. And the goal, you know, the job of the kernel is actually to make bad actions impossible, not just unlikely.

And so you have these two boundaries with the, between agents and the external world. The first one is typed tool calls, the tool calls. You don't want to, you know, you don't want to call tools that don't exist, et cetera, et cetera.

And also the boundary of the other agents, which is typed events. And this is non-negotiable, I found. Like you can get a lot of errors just from this. I wrote a really long blog post about this. It's, if you follow the QR code, you'll find it.

And yeah, and the result of that is I actually deployed it within the company after I built this. And now today, after a month of deploying it, we have 20 agents on the left that are not just contributed by technical people, by the way, which is kind of what markdown, what markdown gives you.

And then on theright is, you know, we deploy, it's called the intranet. There's the briefs, there's a bunch of, I mean, there's a bunch of things as you can, as you can see. Kind of like a few, you know, as a conclusion, a few lessons.

The first one is that well-executed background agents are really magical. They feel like this, you know, robot mower that I had at the beginning is I really just sit down and I come back and I have this morning brief that is probably even better than what I would have had just doing it manually.

And it just appears in my inbox every day and processes my, you know, random thoughts. The difficulties that you meet doing this kind of thing is just good old engineering problems. I mean, there's really nothing new under the sun when it comes to orchestrating these things.

It's just good old software orchestration. Open source models are there. They're good enough. I replaced, so I don't have any third party APIs anymore. Now I just use open source models. And even on my laptop, I use a local model.

So it's good enough for what I do with it. For coding, I don't know, but for what I do with this, it's good enough. The infra category is definitely unsettled. I tried a few things before I started building myself.

And I would advise that today, like definitely start building before you buy. So if you're a small company, if you're a tech CEO, it's kind of an advantage because you can just do this without, you know, tasking engineers to do this and get them off track.

But I would definitely try to build before I buy just to know exactly what I need and, you know, the limitations of what exists. Also, I will say that to people building frameworks for this is please eat your own dog food.

Sometimes it's pretty clear that people are building, you know, agent orchestration frameworks, et cetera, but not eating their own dog food. So please do. And the other thing is I'm really glad I took this two weeks off to play with the field because that completely changed the, I mean, that changed the trajectory of the company.

I know we're an AI company, we should be in it, et cetera, but, you know, business is such that you're always thinking about the next thing, the next thing, the next thing. And it's the same everywhere. But what I'murging you to do is to stop and actually immerse yourself in this and try to see how useful it can be for your company.

So you can still do code. It's not a product that we sell and we don't intend to sell this. You can read our blog as well. So I haven't explained this yet, but I will publish something about it.

And thank you for your attention.