# Local Agentic Theory For Mobile Games — Shafik Quoraishee & Joanne Song, The New York Times AI Engineer · 2026-07-23 Shafik Quoraishee and Joanne Song of The New York Times argue that on-device AI agents can transform mobile game accessibility by tuning difficulty and assistance in real time as a single continuous dial rather than separate toggles. They demonstrate a Space Invaders agent that perceives, predicts, and dodges entirely on the phone within a 16 ms frame, and a mini crossword solver using constraint backtracking. The pair explains the on-device budget—space for weights, time within refresh cycles, and energy drain—and shows how gaze estimation and tap analysis let the agent resize controls or break keyboard traps dynamically. They ground their approach in WCAG accessibility standards and propose a future where billions of local brains, each personalized to a user, replace centralized cloud AI. ## Questions this episode answers ### What are the key resource limitations for running AI agents locally on a smartphone? Shafik Quoraishee outlines three constraints: space for model weights and state history, time to plan within each 16-millisecond frame at 60Hz, and energy to avoid excessive battery drain. Exceeding the frame budget causes visual jank, and current mobile NPUs are not yet optimized for agentic workloads, requiring careful curation of the agent loop to minimize processing. [8:27](https://aiengineer.podhood.com/8cb6a041-e5b7-444d-9244-5752a0749acf?t=507000) ### How does an on-device AI agent play Space Invaders without cloud support? Shafik Quoraishee demonstrates an agentic loop that runs locally: the agent perceives the game scene, predicts alien movements, decides whether to dodge or shoot, and acts accordingly. This perception-predict-decide-act cycle repeats continuously, leveraging on-device models to avoid cloud latency and adapt dynamically without retraining, unlike traditional reinforcement learning approaches. [7:00](https://aiengineer.podhood.com/8cb6a041-e5b7-444d-9244-5752a0749acf?t=420000) ### How can on-device AI improve mobile game accessibility in real time? Joanne Song explains that on-device models track eye gaze to detect search friction, analyze shaky taps for motor challenges, and monitor focus paths to break keyboard traps. The agent dynamically resizes controls and adjusts input tolerance and step granularity, merging accessibility and difficulty into a single continuous dial that tunes the game to the player’s live needs. [14:58](https://aiengineer.podhood.com/8cb6a041-e5b7-444d-9244-5752a0749acf?t=898000) ## Key moments - **[0:00] Intro** - **[1:55] History** - **[2:49] On-device AI** - [2:54] On-device AI agents eliminate cloud round-trips, reducing latency and keeping data private, says Shafik Quoraishee. - **[4:56] RL to agents** - [6:25] Agentic AI systems reason via in-context learning without changing model weights, unlike reinforcement learning, explains Shafik Quoraishee. - **[7:00] Space Invaders** - [7:01] Shafik Quoraishee demos an on-device agentic Space Invaders that loops through perceive, predict, decide, act. - **[8:19] Device Budget** - [8:56] On-device agents must plan within a 16-millisecond frame to avoid jank, highlights Shafik Quoraishee. - **[11:15] Crossword Agent** - [11:15] Shafik Quoraishee demos a constraint-satisfaction agent that solves the NYT mini crossword by backtracking with a constraint graph. - **[12:33] Accessibility** - [13:22] Joanne Song identifies crosswords' accessibility failures: tiny targets, invisible highlights, and rigid timers. - [14:12] Joanne Song proposes merging accessibility and difficulty into real-time dials tuned by an on-device agent. - **[14:40] Live Tuning** - [15:18] Joanne Song shows an on-device agent detecting a keyboard trap and injecting an exit route for keyboard-only users. - **[16:28] Billions of Brains** - [17:28] Joanne Song: 'The future of AI can be billions of small local brains, each shaped entirely by the individual it serves.' ## Speakers - **Joanne Song** (guest) - **Shafik Quoraishee** (guest) ## Topics Autonomous Agents, Inference Optimization ## Mentioned The New York Times (company), AlphaGo (product), AlphaZero (product), Crosswords (product), EfficientZero (product), Google Sima (product), Space Invaders (product), Whirlbot (product), mini crosswords (product) ## Transcript ### Intro **Shafik Quoraishee** [0:13] Awesome. So, yes, as a great introduction took us to, we are going to be talking about local agentic theory for accessible mobile games. And we realize this is the graph track, but there are graphs here in this presentation, so fret not if you're missing them. So, yeah. So, a few disclaimers before we continue. Since we do work at The New York Times, and there are a few things we have to say about our puzzles at AI, et cetera, our puzzles are made by people. They're not made by AI. So that's just a thing that we it is true, and it will always be true. There's no AI in the games themselves, so there's no AI features. You might have seen Whirlbot. That's not an AI feature. So all our games are pretty much AI-free. And the work that we're going to be talking about now is experimental work. So it's stuff that we look at in terms of solvability and other capabilities of agents in this space that we can potentially take advantage of on a local device to do other things than actually generate games, more so things about playability and other kinds of features we can utilize agents for in an intelligent way. So, introdu a little bit of an introduction. So I also, if anyone has seen the last year's World Fair talk that I did on connections, there's a similar vibe to this. We I did a deeper dive into how a solver can be built for that game. And if you're interested, you can take a look. It's on YouTube, and it has a lot of interesting deeper dive specifics for RL solvers, which I'll go into in a bit as well. So, a little bit of a history. Everybody here, and just in case you aren't familiar with AI in gaming, which for, you know, it's not the most common topic here at the World Fairright now, but if you are familiar with the history of it, AI has been in games for a long time. ### History **Shafik Quoraishee** [2:08] And since this is since the '80s, it's been established as a core pattern in games like Pac-Man. This is a simple version of an AI called a finite state machine. And if anybody everyone's familiar with a finite state machine here? Probably. Or I would imagine enough people are. It's a basic AI system which tells the ghosts in the game what to do when Pac-Man gets a dot or not. But basically, it is a conditional kind of symbolic AI. And then, moving forward, we'll talk more about RL and the advancements of where that went to. But firstly, the core thesis of what we're going to really focus on is the model of running the devices, running AI on our local devices. ### On-device AI **Shafik Quoraishee** [2:54] So most AI infrastructure today runs really on the cloud. And if you're running a mobile application, most practical scenarios involve cloud architecture. And that essentially is expensive because you have AI calls and inference calls that you have to make upstream, and then they have to come to your device. So there's things like latency, et cetera. So in an ideal world, when we evolve to that place, we can offload a lot of the intelligence onto a device. What that does is make a lot of the make a lot of the calls faster when we get to that world because you're not doing a round trip to the cloud to get that information. So you can actually compress that time to the time within the device frame computation. It's private, too. So your AI computation is with on it's within the device itself. And if it's set up properly, it won't leave the device because there's some things that, for the gameplay experience, that you don't actually need it to leave the device. Local computation is the way to go, and it stays within your confined security zone locally. And it's not needed upstream for telemetry or anything like that. And then it should work anywhere. So one thing is that games, you know, some a lot of games rely on the internet, and many they will for a live connection point. But at least in the in many cases where you need advanced AI compute, you can develop games that can work locally on your machine, on your mobile device, rather. And that's great because if you're in a subway tunnel or something like that, you can still have a game with a very smart AI that is building doing something without being disrupted due to HTTP calls that are not getting through. And then it's, again, personalization is a big thing in general with on-device models because personalization is important. And a lot of people want their game experience to be tailored to them. And in a world where we can do things on-device, we have that capability more so because the features are local. ### RL to agents **Shafik Quoraishee** [4:58] So now we're going to talk about agentic AI for games. And as I'm sure everybody's familiar with AlphaGo, AlphaZero, et cetera here, the Alpha series models. So basically, reinforcement learning was the way that games were, in the past, trained in order to do intelligence,right? So basically, what you would do is that you would take a you take a model, have many iterations, and then that model would then be really tuned to a particular game. And that's still happening. But what I'm doing is setting up the kind of history and the school of thought of that. All the way up to 2024, has anybody heard of EfficientZero before here? If not, it's fine. EfficientZero and EfficientZero v2 are the reinforcement models reinforcement learning models that are state-of-the-art. And they're called sample-efficient reinforcement learning models because they learn very fast with a lot of a lot less data than traditional reinforcement learning models do on games. And there's just so that was up to 2024, and that mechanic is continuing, but it lives alongside our reinforcement learning and agentic game development. So reinforcement learning, again, is on the left, is a bunch of cycles that are used to train a model in order to change its weight so it becomes more intelligent and able to play a particular game. An agentic system is one where you actually have a game that is negotiated by agents, which are essentially language models, which are using in-context learning to figure out the state of the world in the game. So RL changes the model itself and changes the weights in the model. The agentic system reasons over space in the game, and there's no reward system to grind out. The loop, there's an agentic loop that's at play, which then learns the system through tool calls and other things that are local to the device and using local device functionality and adapts to a new situation. ### Space Invaders **Shafik Quoraishee** [7:01] So it's much more dynamic. So this is basically one example of an agentic-based game using everybody's seen Space Invaders before, I presume? OK. Well, if not, well, there it is. So this is an agentic version of Space Invaders, which is designed to use a local agentic framework, which we worked on to do basically basic things like kill the spaceships as they get close and avoid the bullets as the spaceships attack. And you've probably seen simulations where RL models have been trained to do the same thing. But this is specifically an agentic system that we built to actually play this game. And then you can see the agentic states on theright. And so this is the inside the agent itself and how the loop works. So basically, you have very few basic states where you're trying to perceive what's happening in this where the agent is trying to perceive what's happening in the scene, where it's trying to predict the next move of the spaceships attacking it, where it's deciding what to do next because if the bullet is too close or if the spaceship is too close, it will take it'll make some sort of action, and then it will act in that way, in the appropriate way, and then loop again to see the state. So that's a very simple system, but it's basically a simple agent that's using an on-device model to actually execute its steps. So some challenges in this space are space is the health is a challenge because on the device, you have a limited amount of resources. ### Device Budget **Shafik Quoraishee** [8:27] You have a device budget where you have to fit the weights. You have to compress the state history of the agent. You have to compress the working set or the planning artifacts of the agent, if you even think of the way that an agent will create the way that it navigates a system. And you have to have enough render headroom in order to actually render the game and play the game itself. So this is basically that. And then you also have a time constraint because time itself is budgeted in a particular way so that you're essentially using like a refresh rate of 60 Hertz. So in each frame that passes, you have to which you get a new frame every 16 milliseconds a pound if your refresh rate is 60 Hertz. You need to actually fit the agent planning within that. And if you go outside of that budget, what'll happen in the agent execution, you're going to see jank and other things on mobile devices that are very hard to pull out. And then you have energy constraints. And energy constraints are basically battery life. So battery life, you know, the basic devices these days are not optimized to really handle agentic workloads. Yet, they're getting to that point where you have NPUs, neural processing units, and AI chips that are specifically designed to handle that inference. But at this point in time, the agentic design of the loop on the device has to be very curated to do as minimum energy processing as possible. Otherwise, your phone's die fast as it is. And they'll die even faster. So essentially, there's a constraint function in of this, for example, constraint graph that takes into account all three of these areas. And so there's different ways of managing the constraints in this constraint graph so you can actually make sure that your agent is performing well. So you can handle space constraints, time constraints, energy constraints. And there are some techniques that we use to optimize those constraints so that we're not we're penalizing over focus on one area, and then we're using, like, soft constraining on another area so that we're not penalizing too much if we need a little extra space. But if we use too much time, we want to penalize that because we don't want the agent to disrupt the user experience. So we look at things like gaze estimation models as well because gaze is another new factor. It's been around for a while, but it's evolving to the point where it can actually be useful. And an agent has to take into account gaze model data, which is basically a convolutional neural net which takes gaze data and then can translate it to agentic behavior. And so it just begs to mention that we need to consider things like the ARC AGI score because we also have the base intelligence of the model. And the base intelligence of a lot of these models aren't capable yet of being fully cognizant of all the complex decisions that an agent can make in the most intelligent and possibly interesting games in the future. But we have to essentially surmount this to get to that point. ### Crossword Agent **Shafik Quoraishee** [11:18] And the last thing I'll show you before I hand it over to Joanne is this is another agent that we built to solve the mini crosswords. So if you play the mini crosswords, this is called a constraint satisfaction agent, which uses a constraint satisfaction graph to figure out how to put the words in the crossword puzzle at theright place so that it fills the puzzle properly. And it backtracks in the puzzle and finds theright word, sometimes finds the wrong word. But the constraint graph here is optimized so that if the constraints are getting too crowded or too incorrect, then it will backtrack efficiently. And the agent can use natural language from the graph to backtrack and find the correct crossword crossword layout. So basically, where are we now? We have a lot of different models that we are considering. So we have generalist models, like, for example, Google Sima, which will be the agentic framework an agentic framework that can do a lot of different things in this space. We have agents that do visual estimation on the phone. And then we have entire generative worlds that are being built using AI. And that will be a whole different interesting agentic space to consider. Now I'll hand it over to Joanne to talk about accessibility. **Joanne Song** [12:31] Is this thing on? Yes. Let's talk about accessibility. How can we build a system that truly serves everyone? Like Shafik mentioned, traditionally, games live in a world of fixed state models, rigid, hand-authored. A player's choices often limit to a toggle or a static menu. ### Accessibility **Joanne Song** [12:48] The flaw here, as you could have guessed, is that it's entirely deterministic, predictable, and completely blind to the real-time needs. But we don't need to reinvent the wheel here to battle to build a battle-tested accessible game. We ground our design in an existing international standard, WCAG 2.2. This framework evaluates interactions across four human-centric pillars. Is this experience perceivable, operable, understandable, and robust, giving us a proven global foundation to build our dynamic player states? But you can't have rules without breaking them. Some common pitfalls where games break these standards a mobile game like Crosswords as an example. There are motor challenges we come across where tiny grid cells can lead to misstaps, active cell highlights are invisible to low-vision players, cognitive barriers can be forcing manual logins or timers that are unforgiving and alienating to process speeding to processing speed needs. A static easy mode can't fix a crossword grid that remains fundamentally blind to our players. Although it's still in draft, WCAG 3.0 spoiler alert is following the trends of dropping the binary pass/fail to a graded bronze, silver, and gold scoring. Accessibility is developing into a more diverse scale and moving away from our standard checkbox. The same shift should apply for gameplay. This scale visualizes the values that can change depending on the needs of the player. Here we have the core dials that scale depending on the player's real-time needs. If we tune for a player needing more assistance, we have input tolerance that controls the flexibility around physical constraints, making interactions more forgiving. The step granularity adjusts for the number of operations in a sequence, helping users break complex tasks and tuning to players who need more nuance. ### Live Tuning **Joanne Song** [14:41] The agent's time the agent's entire job is to dynamically calculate these dials as they rise and fall together, balancing the scale against the player's live context. With on-device models, the agent can get a real-time feel for how the player is doing. For example, we can use vision models to track eye gaze and figure out when the user might be facing search friction, analyze shaky taps for someone having a hard time with controls, meet players where they're at by allowing handwriting recognition, focus traps. But we don't want the agent to just watch. We want it to act. We simulate on theright well, the simulation on theright is a broken dialogue, forcing a user into a keyboard trap. For a user using a switch or a keyboard instead of touch, focus order is very important. If we have a broken dialogue, tab and focus can be an endless loop with no way to navigate back. Here, the agent can monitor that focus path, flag the loop, and inject an exit route live on the device. When you're navigating with limited dexterity or just fat fingers, tiny targets can mean a miss. Instead of using players instead of making players play operation, our agent is a live layout auditor. It measures the interface on the fly, catches violations, and dynamically resizes the controls. The agent rewrites the layout live, adapting to the human and not the other way around. Think of it like a real-time tug-of-war where the system is no longer fixed but tunes the game to the player. Accessibility and challenge stop being treated separately and become two ends of one dial, constantly tuning for the moment. ### Billions of Brains **Joanne Song** [16:28] Ultimately, this is where on-device AI is heading. By building systems that can actively sense human frustrations, we can move past fixed menus and turn our devices into responsive, empathetic partners. But there's still groundwork to be laid. Looking ahead for local agents to truly understand games, there is still a lot we need to build. First, they have to be faster. We need a plan. We need plans and decisions within a 16 millisecond frame to prevent stuttering for games. We need models to predict the game so that we can see exactly what a layout change will do before they actually make it. And the real magic, though, will be the long-term memory, learning one specific person's unique habits and needs over time. We need a shared game state language so one agent can work across multiple games instead of being rebuilt from scratch for different releases. And finally, we need better chips and honest testing, faster chips paired with real benchmarks to prove that the agent makes things better. And the future of AI doesn't have to be one giant centralized brain. It can be billions of small local brains, each running on a personal device, each shaped entirely by the individual it serves. Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.