# Wearing the Agent: From Group Chats to Glasses — Sai Krishna Rallabandi AI Engineer · 2026-07-29 Sai Krishna Rallabandi argues that group and wearable settings force agents to be redesigned around shared memory and security, as single-user assumptions break in multi-user contexts. He shares eight months of deploying Jodith among friends and family, highlighting two core challenges: guarding and memory. On security, two individually safe skills—OCR and reporting—can collide at runtime, leaking PII 90% of the time; his defense is a deterministic guard at the action surface and a LoRA fine-tuned SLM that catches prompt injection even when characters are obfuscated with dots. For memory, he proposes a continuously adapting relevance scorer to compact context and graph-based retrieval for evolving group conversations. Finally, he advocates per-user LoRA adapters on a shared memory layer to bake in privacy permissions instead of code-based access control. ## Questions this episode answers ### What security risk emerges when two safe agent skills share infrastructure? In group agent settings, a paper found that two safe skills like OCR and reporting can collide at runtime when sharing infrastructure, causing the reporting agent to leak personal data. This affects about 90% of attacks. The defense is to read everything but guard at the action surface, using a fast deterministic system and a fine-tuned small language model with per-user adapters, catching prompt injection that regex would miss. [7:44](https://aiengineer.podhood.com/0462d159-f2a2-4808-aa0a-ec4716b57107?t=464000) ### How should memory be designed for an agent operating in a group chat? Unlike single-user agents that can compact conversations naively, group agents need to extract atomic facts from discussions and use autoraters to continuously score relevance. A relevance scorer adapts over time, enabling knowledge-based compaction to save tokens. This avoids bloated context and ensures the agent retrieves the right information. Additionally, the ingestion engine must be aware of KV caching to prevent breakage during retrieval. [12:29](https://aiengineer.podhood.com/0462d159-f2a2-4808-aa0a-ec4716b57107?t=749000) ### How can a shared group agent maintain privacy across different users? A shared group agent can maintain privacy by using a common shared memory layer but training separate LoRA adapters for each user. These adapters bake permissions via machine learning rather than code, so the same data (e.g., salary) is treated as private or public based on context. This approach, inspired by the 'user as an n-gram' paper, also includes training a model to determine when the agent should speak or remain silent. [16:57](https://aiengineer.podhood.com/0462d159-f2a2-4808-aa0a-ec4716b57107?t=1017000) ## Key moments - **[0:00] Intro** - **[1:23] Group Shift** - [1:23] Almost every agent built today serves a single user, notes Sai Krishna Rallabandi. - **[2:39] Jodith** - [3:57] Sai Krishna Rallabandi's glasses agent spoke privately to him while driving, not through the car speakers, to protect privacy. - **[5:17] Three Pillars** - **[5:56] Security Guard** - [5:56] Agentic systems cannot be secured like large language models because they operate on a broader action surface, argues Sai Krishna Rallabandi. - [7:55] "Two skills which are benign at the surface, when they run together, can be malignant," warns Sai Krishna Rallabandi, citing a recent paper. - [9:24] Sai Krishna Rallabandi proposes letting the agent read everything and building a deterministic guard at the action surface as the first line of defense. - [11:35] A fine-tuned model catches prompt injection attacks using interspersed dots that evade regex-based approaches, explains Sai Krishna Rallabandi. - **[12:13] Group Memory** - [13:13] Extracting atomic facts from conversations builds smarter memory for group agents than storing entire transcripts, suggests Sai Krishna Rallabandi. - [15:00] Training a continually adapting relevance scorer enables knowledge-based memory compaction and saves tokens, says Sai Krishna Rallabandi. - **[16:34] Privacy** - [16:34] A shared agent creates a new social contract where the same data can be public or private depending on context, explains Sai Krishna Rallabandi. - **[18:21] Outro** - [18:21] Sai Krishna Rallabandi concludes that group agents require redesigned security, memory, and routing to handle multi-user settings. ## Speakers - **Sai Krishna Rallabandi** (guest) ## Topics Agent Memory, Guardrails ## Mentioned Cloz (product), Inspect Agent (product), Jodith (product), Nemo Cloud (product) ## Transcript ### Intro **Sai Krishna Rallabandi** [0:14] Okay. Good afternoon, everyone. Uh, I'm Sai Krishna, and firstly I would like to thank all of you for attending the talk. This is the final session of the final day, so I realize, um, thanks you—thanks a lot for attending the talk. So let's get started. This conference has really been about agentic systems,right? Every workshop that we attend, the keynotes, the speaker sessions, the conversations that we have been having in the hallways, all of us—all of us have been discussing agents in one form or the other. And I would like to declare that, you know, we have won. Like, we have built the agentic systems. We can build them. For, you know, a very basic reference, what is an agent? An agent is nothing but a combination of systems: the model, which is the brain of the agent; we have harnessed some form of orchestration around it; and a set of tools that lets the agent do something. Now, this is pretty simplistic. We can build an agent in an afternoon; it'll start doing useful things by the evening. And yes, there are a lot of issues today in terms of, you know, the traces, etc., but we can build systems today. And I would like to project forward, trying to see what comes after this. And let me start by a basic observation, which is that almost every agent we build today has the customer of size 1. Cloz, Nemo Cloud, different flavors of them, all of them cater to one person. ### Group Shift **Sai Krishna Rallabandi** [1:43] Even the programming assistants that we build, even when they are deployed in an enterprise setting, they are typically geared towards one customer. So this is great. We know the problems and challenges with this setup. We know the shape of the things that we want to build. But probably the next agent we are going to build is not going to have one single customer. It's probably going to be serving a group. On top of that, it's probably going to be in wearables such as glasses, and it's going to be on all day. So one of my advisors used to say, you know, you work hard as an engineer to solve a problem, and then the moment you solve it, you realize that the question itself has slightly changed. Maybe we are heading towards a setting like that, where all of our engineering, we have built it for agentic systems targeting one customer, whereas we might be entering an era where we are deploying these agents in group settings. And group settings pose uniquely different challenges compared to settings where we have single users. So I have been working, uh, inspired by this, um, with an agent called Jodith, which is deployed in a group setting among friends and family for a period of 8 months. ### Jodith **Sai Krishna Rallabandi** [2:59] Let me illustrate what I mean in the talk by taking a few examples, uh, taken from the production system itself. On the left, we see one example from this week, where it's deployed in a group of attendees of this conference, and I was asking it, okay, how do we go to the conference venue? And the agent didn't—chose to not answer in the group, but DM'd me because of the privacy issue. Similarly, this is a conversation from a couple of weeks ago, where my wife and I were trying to organize an event, and it's tried to sync up all the calendars and, you know, make sure we get a slot which is available for everyone. Then there is a proactive aspect as well. All of us, I think, after the, uh, release of coding agents, we are operating on pretty light sleep. And the fact that the agent understands, uh, we are under less sleep and, you know, we have, um, you know, trouble decision-making, that's not impressive. The impressive part, uh, when a glasses agent spoke to me this was, I was driving, it didn't—it chose to not announce this from the media of the car, but it chose to speak it directly to me in the glasses, preserving the privacy again. And then, um, a group setting has conversations that are lasting over a period of time. So this is an example where a bunch of friends have been discussing a particular topic that is constantly evolving, and the agent was, uh, intelligent enough to curate the content and the memory which is relevant and filter out the things that are not needed. And finally, a simple application from an agent that my daughter uses, who is 3 years old, who is using the agent to, you know, learn a lot of things, like capitals of the countries, different, uh, numbers, etc., keeping us posted as well as to the progress of the kid and making sure that she doesn't forget the things that she learned and we also are aware of the things that she's learning. So all of these are examples where agents in a group setting have slightly different dimensions when they are deployed. And there are—the aspect of group deployment is also not unique. There have been attempts in the last 6 to 8 months of trying to deploy agents in a group setting under controlled scenarios, try to try and see, you know, how they fare, etc. ### Three Pillars **Sai Krishna Rallabandi** [5:17] So leading to the recognition of, uh, the growth of agents in a group setting, my talk today is going to be focused mainly on three aspects in group chat agents. One, the guard of the agent, which is what—what lets in the memory of the agent. Today, in agentic systems, we know that the harness involves a lot of orchestration in terms of the memory, but group chats take it to the next level. And then the routing. The routing here is not the model routing to save tokens, but routing of the information, the end information to the final user. So let's get started with, uh, the memory, the security layer, which I've—I'm calling Jatayu. It's a mythological character from Indian mythology. The tagline from here is that we can't secure an agentic system like we secure a large language model. ### Security Guard **Sai Krishna Rallabandi** [6:12] What I mean by that is, a large language model, when it hallucinates, we can inspect the output and di—design stuff on top of it. But an agentic system, by nature, operates on things as well in addition to the large language model, which means the surface area that the agentic system touches is much more richer and more vast compared to a large language model, and it is amplified in a group setting. So imagine this,right? When we have an agentic system deployed in a—even in a personal setting, it's going through every web page that, uh, is available, and the attackers, who can do prompt injection and various other types of attacks, which I'm going to talk about, they have access to it all day. Every group message, even though the people in the group, we have added them, but imagine a scenario where somebody takes a screenshot of something and uploads it. Uh, every GitHub issue that we post and track updates on, there have been a lot of cases of late where, you know, tokens have been exploited that way. Every email that we get, the promotional emails, etc., the agent reads them, and all of these are surface areas where the agent can be exposed to, uh, vulnerabilities. And therefore, we can't guard everything and remove everything from entering. There has to be a balance between how we design the security layer, and which is why I'm saying it's different from the security layer when it comes to typical large language models. So to illustrate this, one of the recent papers which came out is, uh, it highlights a very interesting, uh, phenomenon where the punchline is that we can't read our way, or we can't model, uh, check our way to safety. And what we see here are two separate skills in the context of a typical large language model. One of them is an OCR module. The other is a reporting module. Both of the skills, static scans, are pretty good. They're pristine. They pass them. But the paper Runtime Skill Audit fo—found out that a static scan surviving code can break at runtime. And the paper, when safe skills collide, also identified that two skills which are benign at the surface, when they run together, they can be malignant. For example, uh, imagine the skills are sitting in our infrastructure, but the—the attacker is attacking the content that the agent is reading, which implies that the OCR extracts the information, but the reporting agent, when it sends the information, along with the information, it also sends our PII to a third party. To put the numbers to the story as well, this is not something which happens, um, at, uh, at low statistical frequency. The papers have observed that around 90% of the attacks have this. So if we want to think about safeguarding our multi-agent, multi-group setting architectures for this, probably the first level of defense has to come at the boundary. Instead of guarding whatever the agent reads, we let the agent read everything and then design a guard which is deterministic so it's fast. We don't have issues with respect to latency. But design when it is taking the action. So for example, when it is reading bash variables, or when it is exporting something, or when it is reading secret variables with respect to configs, etc. And then, since we can design a deterministic system using traditional NLP or regex-based approaches, we can flag these out, saying, hey, some of them are benign, so let's allow them, but some of them need approval, etc. So this is the three-class categorization that we can come up with as a naive approach. Now, uh, we all know that this approach is good, but we can probably do slightly better, which would be to say, can we train a model on top of it as well? So within Jodith, there is a component which tries to look at this data and tries to build a supervised fine-tuning model, an SLM, which is LoRA fine-tuned, to try and predict if there is something malignant in the data channel. Separate the data channel firstly from the instruction channel and only give out that information which is purely benign. So the ideal response here would be if the instruction is, summarize this page for me, and the website that we have crawled contains some prompt injection attack or any other form of attack, to obscure that information and the model just performs the task as required. So, uh, we have also benchmarked this on Inspect Agent. That's one of the benchmark, uh, datasets available here, and it's—it's the naive approach here is basically at 50%, which is a coin flip, and it improves on top of that. A very interesting caveat here is that a particular, uh, attack was spotted where instead of writing the text normally, if you write it interspersed with dots, like i.l.i.k, the regex-based approaches and most of the static approaches fail at that. Whereas a learned model, uh, would probably catch it, which is where the attempt to build a learned model comes in. But to summarize, when we are building a, uh, security layer for an agentic system, be it personal or be it group-based setting, we want to be able to build a system not at the input, because that is going to gate everything, but at the action surface where the agent actually is performing some task, and maybe a better approach than a naive approach like regex or traditional NLP is to try to build a model against which we can do some hill climbing later. ### Group Memory **Sai Krishna Rallabandi** [12:13] Once the input comes into the model, the next important interesting thing when it comes to group, especially when it comes to group settings, would be how do we organize the memory. Because in a traditional setting, in a traditional model, uh, the memory implementations can be very naive. A memory implementation can be done just using embeddings or files. But the interesting, uh, thing to note is that the model is just the engine, especially when it comes to a group chat. When we deploy it in either friends group or in a work setting, the memory is what the agent becomes, and it's becoming in real time. It's evolving in real time. So we have to be very careful when it comes to how we store things, how we curate the thing, and how we forget some of the things that are stored in memory. It's—it has applications with respect to latency, because the less we store, the better token usage and everything else. So I would like to talk about, uh, five aspects here, uh, briefly mentioning, uh, each of them. The first one is what do we store? Let's take a typical conversation, um, like that happens in a group chat setting where people are discussing one particular occasion, where the first person says, you know, hey, um, let's go to this place, and this is the day at which we are going. Now, there is a follow-up conversation. Somebody else adds some more information to it, which keeps happening. Now, one approach which commonly, uh, naively what we do is to store everything and then keep compacting the conversation as the conversation becomes overhead. A probably smarter way would be to ex—extract some form of atomic information, atomic bits from this conversation. In this case, the atomic bits would be, uh, anything that is relevant to the conversation which might come into future. So once we build a model like that, the important thing here becomes how do we, uh, design autoraters for this. And the autoraters have to be designed on extra—did we extract high-value atomic facts or not? Given a conversation, are these the relevant information or not? And then, are these hierarchically related to each other or not? The temporal aspect as well, because some things become more important as the time goes along, and some things we might have to drop. And then, are we retrieving theright set of information given the current query? And then, is there some form of a intelligent retrieval as opposed to crawling everything by organizing information in form of a graph, etc.? So, uh, the other interesting aspect when it comes to memory would be what, uh, what we store and what we forget. As the memory becomes bloated and bloated, we might want to train a very simple machine learning model or an SLM to try and figure out what are the important concept context, uh, information that we need to store. And a typical approach, a very simple approach discussed in the paper Learning What Not to Forget is where they learn a relevance scorer which keeps scoring continually. The important word here being continually, because in a group chat setting or in a group work setting, the context keeps evolving and different things become important at different points. And we can, once we have a relevant scorer which is continuously adapting, we can save much more on the tokens because now we are doing knowledge-based compaction. One other thing, uh, when it comes to serving these stored memories is, uh, whether we use cloud-based serving or if we use local-based serving, we need to be aware of the fact that we all of us use KV caches, and KV caches break when we try to do something cute with respect to the model. Therefore, building an ingestion engine which is aware of KV caching also helps there. So all of these, uh, are the components which are important while designing a memory. Again, um, the memory component here is slightly different from memory from a personal engine, and all of these become sup—uh, extremely relevant, uh, in a group setting. ### Privacy **Sai Krishna Rallabandi** [16:34] So I'll talk about the final part of the talk, which is the privacy aspect. Uh, a shared assistant is not just a large model which is shared between different people, but it's almost a new social contract because the information that is, uh, passed to a shared agent, it's either public or private depending upon the context in which it is presented in. The same information, for example, grocery list is benign for everybody, but information about salary or information about somebody's health becomes extremely private as well as public information depending upon the context. The data has not changed. The room in which it is deployed has changed. So a very neat trick for this would be instead of storing all of the data all the time, take inspiration from, uh, the way human brain works and try to have a common shared memory layer, but train different, uh, LoRA adapters on top of each adapter for each of the users. Therefore, the permissions are baked in. Now, this is baking the permissions not by using code, but by using machine learning itself. The user as a Ngram paper is a great reference, uh, of how to do it at scale. And also knowing when to speak and when not to speak. A typical observation from where agents are deployed in group setting is they tend to be overproac—be over, uh, articulative when they are not asked questions. And a simple approach here, which is currently still in progress, would be to train a model to determine, uh, it could be as simple as a robot classifier, but to determine the role and when an agent is allowed to speak and not speak. So, uh, I would like to end the presentation, uh, saying, hey, when we are developing harnesses when it comes to group agents, we need to keep in mind three additional things at least. One of them being the security layer, the gate which keeps information out or filters out the information depending upon the context, how we design the memory, the harness, and how we route the information to the responsible parties. ### Outro **Sai Krishna Rallabandi** [18:49] Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.