Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

Yeah, so yeah, a bit of a change of speed from the last two talks. So we're looking at kind of higher-end robots. If we want for intelligence to get into lots and lots and lots of devices, and not just really expensive robots, we are going to need tiny models.

And this talk is about what is the state of the art of tiny models at the moment, what are the things they're good at, and what are the things you can go start building today. OK, so yeah, firstly, a bit of background, like briefly on me and the team I work on.

Then we're going to take a look at small models that you may be kind of more familiar with, just kind of explore what they can do, what they can't do yet, and then kind of see, hey, why do we need even smaller models?

And then just looking at the state of the art of tiny models today and what you need to do to get them into a form where you can deploy them in production to do useful things. And yeah, lastly, we've got a couple of examples that we can look at from work from our team.

OK, so me, our work, I've worked in edge AI for a while.

These days, I work as a tech lead on the AI Edge team at Google. Within the team, the types of things we do are we develop kind of open source projects called like Lighter TLM, Lighter Team MediaPipe, and these make it easy to deploy AI to edge devices.

We also do a lot of work delivering kind of edge AI core technology to Google's own products, some of which would be via tiny models. And then we also work with the Gemma team to ensure their models work well and run well on lots of devices.

And then we have a significant focus on small and tiny models because

that's what's useful for a lot of kind of mobile phone applications, or if we want to be able to ship a model in browser, that also has to be really, really small. And generally, our kind of playbook is we develop things for first-party, like for in-house use first, and then if we can figure out a way to share that via an open source package or make those tools available to the wider world, we do so.

And that helps kind of lots of other people build similar types of things using open source technology.

OK, so why do edge AI? This is probably like, as opposed to just doing everything in the cloud, you know, it's kind of obvious, but I'll kind of go through it anyway. There's like kind of latency, you fast, consistent speed, privacy, data stays on the device.

Offline use, it's kind of reliably available, so that kind of

that feature that you rely on in your mobile device will still work even when you don't have reception. That can be very helpful. And then savings, especially these days, if the alternative is to call

even a faster model on the cloud, that will come at a cost, particularly if you're kind of shipping an app or like a mobile phone app or something in browser where the user interaction is at kind of very, very large scale.

Then even though those tokens are relatively cheap, you're multiplying it by a large number, and it'll add up quickly.

So then the main challenges then of deploying AI on the edge is the leftmost one is kind of new, which is DRAM cost. And it's a really significant constraint that and you'll even see some mobile phone manufacturers are putting less DRAM into their devices this year than previously.

You'll also see that since launch, the cost of a Raspberry Pi 3 16 gigabytes has gone up by a factor of like 2.5x. So DRAM cost is really, really significant. That then has a casts a shadow over the rest of this talk,right, where in order to be able to get AI applications running on the edge, we need to really think a lot about kind of quantization.

And we also really need to think about what is the smallest possible model we can use for a given task. Other challenges are, yeah, there's a wider pool of target devices. And yet another challenge is, yeah, it's kind of fair to say that a lot of the research hours that go into LLMs these days are into the much larger models and MOE techniques and this types of stuff.

And the lower end of the LLM spectrum is a lot less studied. So yeah, these are challenges of deploying the edge. OK, so small models. And when I say small, I would mean kind of typically maybe kind of 1 to 2 or 1 to 4 billion parameters.

You may find that these are built into the OS. There's a version of a small model that ships in Android high-end phones today with AI core. There's a version that ships with Apple, with Apple Intelligence.

Some app vendors will ship models this size in their app. We certainly work with some app vendors that do this.

And for like IoT and robotics, you would typically require like maybe 4 to 8 gigs of DRAM in order to be able to ship this grade of model, which then has an implied cost on the device,right? So it then kind of restricts these models to things like laptops, mobile phones, or kind of higher-end electronics and kind of puts it out of reach of maybe a lot of lower-tier web browsers or the wider kind of IoT and consumer robotics market.

Yeah, and for smaller models, yep, developing smaller models, and we'll look in a while. We do a lot of work to minimize footprints with kind of quantization. And the playbook here is mostly prompting,right? If you want to deliver a particular feature using a smaller model, you can just kind of use zero-sock prompting and get pretty good performance.

Also, to use LoRA adapters. And it's kind of somewhat robust at doing things like kind of function calling and agent skills. OK, so really quick example is like our working with the Gemma team. Our favorite go-to example is always Gemma for these kind of things.

So we can see that like the E2B model is pretty capable in terms of reasoning. It's certainly like on par like with a Gemma 3 much larger model from kind of 12 months ago. And yeah, so we now have like much smaller models that are pretty capable at reasoning, and we get pretty decent answers just with zero-shot prompting for a given task.

We've also done lots and lots of work to optimize the memory footprint of that 2 billion parameter model as much as we possibly can. So it uses a mix of like 2-bit, 4-bit, and 8-bit quantization, getting it down to like 2.9 bits per weight if you look at the actual weights we need to hold in memory.

We do other tricks like per-layer embeddings. We won't go into like all of the detail here. But end result is we can you need maybe one like here it's 841 megabytes for a text-only model in memory just for the weights.

And then maybe by the time you add in the runtime, a KV cache footprint, you might be up to requiring like 2 gigs of active RAM to be able to run this model. Then you account for an OS and the fact that there's other things going on.

That's where we get the kind of 4-gig plus rule of thumb for deploying this on a device.

Then in terms of speeds, this is using our runtime. This is just a list of devices that we run on. For the purpose of this talk, we're going to look more closely at the last three rows of the table, which is if we take that 2 billion parameter model and run it on a Raspberry Pi, that will give about 7.6 tokens per second decode.

This is without MTP. If you turn on MTP, that'll get maybe 2x faster depending on the task. If you go to a higher, a more capable device like a Jetson or a Nanoe, we can get up to maybe 24 tokens per second decode, or maybe even faster if you used NVIDIA's own toolchain.

This is with our toolchain. We also have done work to port this to a Qualcomm IoT board, which is pretty popular among kind of higher-end robotics and IoT applications. And there, yeah, you can see you can get about like almost 4,000 tokens per second pre-fill, 31 tokens per second decode.

And that's useful for lots of like almost real-time

applications on an NPU because

with these with Gemma 4 models, like one medium resolution image is like kind of 500 tokens. A high-resolution image is 1,120 tokens. So you could get like three frames per second of high-resolution tokens going through this model and have pretty decent decode speed as well.

So there's lots and lots of compelling applications you can build with this type of with a small model. If you're kind of kind of market or if you're kind of willing to have more expensive hardware and have a more expensive

DRAM kind of line on your kind of bill of materials for the device you're building.

Yeah, just like our toolchain, we also work with other models in the community that are of similar size, and they each have their strengths as well,right? So these are some of the other models that we support here. Really briefly, I won't go into this in too much detail, but we also if I can get this to play.

We also have an app that you can use on both iOS and Android. So if you want to take one of these small models, see how fast it works on a phone, you can just go straight ahead and do that.

So it's available on AI Edge Gallery. Also, all of the

why I'm getting that buzzing. The app is also fully open source. So if you want to see how to build something similar using one of these models or to see how this is using the open source runtime that runs the models, you can see all of that.

So this is a great way of just getting started and trying small models if this is what you want to do.

OK, this is another example, which I'm not going to play this video, but you should definitely check it out. This is an example showing an open source, the open .mini v2 robot. This is one. Xavier, one of the engineers in DeepMind built this.

It's kind of a hobby project. Really, really fun. So go check out this YouTube video. What you'll see is these two robots. One uses the Jetson Nano. One uses the Raspberry Pi. And you'll see that the robot is able to

it's able to kind of like read signs and react to things and kind of nod its head. It's also kind of able to take both voice and image input. Yeah, and what you'll see is the Jetson Nano one performs has really good real-time interaction.

The one based on Raspberry Pi, it works, but it's kind of a lot slower,right? So for some examples, for some types of interaction, even the kind of best models we have today are maybe not meeting kind of user interaction requirements.

But yeah, this is a really fun video, so definitely check it out. So yes, so then small models, while they're great,right? If your product can afford to use one of these, they're really easy to use because you just need to zero-shot prompt in order to get it to work.

Gemma team has done great work in having low-footprint, high-capable models that are ready to use, and they're optimized running on all of those devices you saw earlier. And if

yeah, so if all of your constraints if you can live within those constraints, then great,right? Your journey would stop here, and you would build a feature you would want,right? For lots and lots of other things that we do in our work and other people that we talk to, we're still at a point where small models are too big because they can't reach like older laptops or kind of more consumer-edge devices.

The user interaction needs to be more responsive. We also kind of have the reality, and we do have this a lot of times, where the model you want to run isn't the main feature in the application. It's like one tiny thing in a corner that needs to run while everything else in the system is running.

So we also need a smaller model for system health is a common pattern. So then enter kind of tiny models,right? So these are typically in the as small as kind of 50 billion parameters. We've deployed models that small to maybe 500 million parameters.

They're easier to ship natively with applications. They would run on the types of things you see on theright-hand side

and would require maybe less than 2 gigs of RAM or even less than that. And they can also be made to run really, really fast. But the playbook to deploying here, it's a little more complicated. So sometimes there's off-the-shelf models that'll do what you want, and we'll look at those in the next slide.

Or else, if that doesn't work, you're going to be left in the world of kind of fine-tuning a model to achieve a given outcome, which works very, very well. So fixed task models, there's a bunch of things around ASR, vision, and embedding models.

And if you have a if you have something yeah, so like ASR and vision and embeddings, these are all kind of stock features, and they work really, really well. This is an example of Apple FastVLM, which is a 0.5 billion parameter model running on an Android device using hardware acceleration.

And you can see it runs really, really fast. So if you needed to kind of add a little bit of visual intelligence to a like an edge device or an IoT device, this class of model is an excellent option to get that kind of first-level visual awareness.

Or for ASR, yeah, there's some kind of strong models listed here as well.

And then lastly, embedding models are great at yeah, like this is just a text embedding model, which is, yeah, really good at kind of processing and matching text, which can be relevant in some cases.

OK, but then

next scenario is you want to kind of fine-tune a model. So here you can start with the models I'm citing here are kind of Google developed models. So there's some starting at like 270 million parameters in Gemma 3 and Function Gemma.

Gemma 3 is a general-purpose model. Function Gemma is one that has extra pre-training for function calling patterns. So here the performance, if you remember earlier on the Raspberry Pi, our performance was at mid-single digit tokens per second decode.

So here that kind of jumps up to 45 tokens per second because we need to read less from memory each time. And we can fine-tune this to do pretty compelling things. So on theright-hand side, this is running what we call a mobile actions model.

So this is just text in and function calling out. This model knows about 10 different output functions and can call them at over 86% reliability from a given arbitrary text input. And this is for doing common things on a mobile device like schedule a calendar or turn on and off Wi-Fi or things like this.

And it can take like arbitrary free-text input and convert that to appropriate function calling. And for this demo, we've taken another ASR model and put it in front of that, which gives kind of voice to function calling as a feature.

And voice to function calling is pretty key for lots of IoT and edge devices because,

yeah, like smaller devices tend to have require settings menus, and that user interface can be really, really challenging for lots of people. So yeah, being able to just talk to something to ask for a given outcome, this is a pretty key capability.

And we can do that reasonably reliably using a fine-tuned small model. So the playbook is generally then you pick a base model. You check the performance if the performance and memory footprint are within the range that you want.

And then the kind of the harder part is you the playbook we've found works really, really well as we synthetically generate data to fine-tune that model. Depending on the model, like there's a data set we've open-sourced here called Mobile Actions.

It's available on Hugging Face that corresponds to this if you want to kind of recreate that same demo yourself and fine-tune Function Gemma from scratch. But we've generally found that in the range of 10,000 to 10 million samples of synthetically generated data will be sufficient to fine-tune a smaller model to a really, really high degree of reliability.

And so for other tasks we've done, like things like summarization or proofreading, so something which you could do with a 2 or 4 billion parameter model reasonably reliably, if you're willing to put the time and energy into creating a synthetic data set and fine-tuning a model, you can achieve a similar like the same or greater quality with a model that is much, much smaller, will work on a much wider set of devices, and will be much, much more responsive.

So yeah, and that's the type of outcome we're seeing now with just fine-tuning a model for a single task. And it's a really like yeah, we found this is a really good playbook for deploying at like very wide scale.

So here's another example in this is one example in production where we have this is an app that we've developed for voice dictation without subscription. All of the voice dictation happens locally on device. And as well as just doing dictation, it also does well, it kind of cleans up ums and ahs,right?

If you see on theright-hand side, it's able to clean up text. It's also able to do biasing towards kind of words and names that are kind of relevant to you personally. So kind of personalization. The left-hand side kind of shows how we built that application.

So there's an ASR engine and a text policing engine. And both of these are fine-tuned versions of tiny Gemma models. And this allows us to take something that would have been a kind of like server-only feature of where you require a subscription to do highly accurate voice dictation and have an app that's just able to do that completely offline with very, very good quality.

So this is something you can try on iOS if you want to give this a go today. But yeah, and it just the backbone of this app is kind of two fine-tuned small Gemma-based models in the low single digit hundreds of parameter million parameters.

We also worth noting is there's also kind of features in developer preview in Chrome, for example, that kind of summarization and proofread APIs are feature as built-in APIs in Chrome. And delivering those features via tiny models allows the Chrome team to ship them to a much wider set of Chrome users than would otherwise be possible.

Yep, so that's we've probably got to have like one minute for questions. Some kind of key takeaways on the last slide if I can get there. Yeah, so takeaways from consumer devices and entry-level robotics is small LLMs are easy to use.

And especially on NPUs, they're very, very fast. Tiny models will enable reach to a much, much larger pool of devices. And voice to function calling can now be built to be robust using tiny models. It just requires kind of investing in an appropriate synthetic data set with enough samples.

And then you can fine-tune a model to get really good outcomes. Cool, so happy to take one or two questions or if anybody has one. Yeah?

Sorry, I'm going to plug this out. Yeah, sorry. Sorry, say again?

Sorry.

Like broader ambitions of where tiny models can go? Well, I think kind of generalizing voice to function calling is one key goal, like making that very easy for lots of people because I think that's a key use case that if we can figure like if we can figure out how to make have like an agent generate the synthetic data for you,right?

It's certainly possible to make that journey much easier than it is today and make it available to a lot more people.

Yeah, and certainly the visual input as well. That takes a little bit of time at the moment. There's certainly scope to have faster models there that can do a wider set of things like kind of segmentation and other things that would enable other use cases.

Awesome. Yeah, due to the time, we probably don't have a Q&A session for today. Yeah, but Cormac will stay after the session maybe, and you can ask four more questions about.

I'll stay after the session, or you can come grab me downstairs at the DeepMind booth at 4 o'clock. I'll be there 4:00 to 5:00. OK?