Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax

Joining us on stage is the co-founder and Chief Science Officer at Hugging Face, Thomas Wolf.

Hello everyone. Hello, Olive. Nice to have you on stage.

Hi. Nice to meet you. Thanks for having me, yeah.

So I think you're on for a treat today, because you just saw GLM, which is current number 2 on the artificial analysis table. I take that table out, because nobody can use it. And now we have number 4.

So basically, you will have all the top models, at least the top open-source models, in a row. And we're very lucky to have Olive, who has a pretty amazing path in life. So she came to the US, Pennsylvania.

She was studying, doing PhD at NYU, in the lab of Yann LeCun, working on JPAR. But we decided we won't talk about JPAR today,right? Something for another day. And then, instead of joining Hugging Face, which was in New York also at the time, she decided to go join MiniMax.

So for those who maybe don't know all the neolabs around the world, and you're forgiven, because I think there's like 64 neolabsright now, MiniMax is one of the top of what we call the AI Dragons in China. So these are the new there's DeepSeek, which is very well-known now, Moonshot, Huda's Kimi, Z, and GLM that you just saw.

And now we have MiniMax. They're all extremely good, extremely talented team, fighting for the first spot. So the latest release of MiniMax was M3, just earlier in June, which was the top model at the time, top open-source model.

Very impressive. There's a lot of very interesting things about these models, so we'll quickly dive in them, and then talk a little bit about what's specific about MiniMax, what's great there. So maybe, Olive, to start a little bit, can you give us a little bit of your view of M3, what you like about this model, how was the release?

Yeah. M3, we released M3 earlier this month, and it is a smaller model with around 400 billion total parameters and 20 billion activated. But it is very capable in terms of both coding performances, and also it understands vision.

So that's what open-source models don't usually have, is that they can the model can not only deal with coding, but it can also understand videos, images, and it has a super long context with one million, with our new architecture called MSA, MiniMax Sparse Attention.

So we really put these three things together, because we know that they will be very important in future AI applications: coding capabilities, agentic capabilities, longer context, and multimodal understanding. Yeah, I think that would be very interesting about the model.

Yeah. So there's a lot to unpack in this model, and it's still, I think, the only top five open-source model that is actually multimodal. So we need to talk about that. But maybe first about the long context, because there was also the first one that really had this real one million token long context that's actually functional.

And you guys had also the MiniMax sparse attention, which is this one technique to make that efficient, that you also published and share extensively. So can you talk a little bit about this, maybe how the project went from the attention, how to make this long context?

Yeah. I would say the story about long context went back to even MiniMax M1 and MiniMax 01, where the model was actually able to perform tasks with 10 million token context.

10 million.

10 million, yes. But then it was not an agentic model,right? It was just, for example, dumping a book, it would be able to give reviews on it, stuff like that. So what we realized was that longer context actually unlocks a lot of capabilities, especially when interacting with users.

And now when the agent is interacting with the whole environment and getting all the tool responses, getting multi-rounds,

shorter context wouldn't be enough to perform the complex tasks. So for this version, we said, oh, we have to have our longer context back. So what we pursued was with our MiniMax Sparse Attention, which was the architecture that was scalable and had a simple design.

So I would say, from a higher level, it has an index branch that selects, on a higher level, what matters more in the context. And then we have a sparse attention branch that performs the calculation on the selected blocks to actually perform the tasks.

And so, yeah, like that, we really designed an elegant architecture so that we can scale the length, and then scale the model size in the future with that.

That's beautiful. I like how, for those who've been in the field for quite some time, we had a lot of work on attention,right? This n-square, and there was a lot of linear attention.

Yeah.

And then somehow all of this disappeared at some point. When FlashAttention came around, we discovered we just needed a more efficient kernel. And now I like how we come back to thinking, first principle, what is attention? How can we make that more efficient?

So one million token is crazy,right? GPT-2 was 1,024, and everyone was like, oh, that's really big. We'll never need more. Where do you see this coming, going in the future? Like Jeff was pitching me the other day a trillion token attention.

You think we should go to token attention?

That's definitely something we can explore towards,right? Ultra-length of context, definitely. That's something that's very exciting to explore with, and something that architecture design, along with hardware, would require a lot of research onto that. Yeah.

You think there's still a lot of low-hanging fruits? So typically today, we saw OpenAI really reducing I mean, we don't know how, as a firm, but reducing their inference bill by half, by probably having some more efficient processing around attention.

Do you think there's still a lot of low-hanging fruit that can be getting, how we can process that? So one thing still very interesting about M3 is how cheap it is, in particular because of this part, attention, or in part because of it's small one, but it's also very efficient.

Right.

You think we can go even way further? Maybe how did you guys invent MiniMax Sparse Attention? Was it an agent coming up with the idea? Was it a human still coming up with the idea? Tell us a little bit about it.

Yeah. So we do think there's still a lot of work that can get into architecture and inference optimization, so that the model can be more efficient, especially if there are tasks that are very task-sensitive, but require very strong capabilities,right?

And for those kind of tasks, we really want the model to be efficient. And who came up with Sparse? Actually, I think an intern from our team worked on that.

Nico?

Yeah, an intern. That doesn't usually happen in a lot of labs, because I think in some labs, interns don't have access to the data, the work, and stuff. But yeah, we are open to anyone who would like to contribute to our model.

So the architecture was actually designed by an intern.

That's very good. Still some work for interns here. Good news. That's also a good segue to also how MiniMax is working internally. So we were discussing before coming on stage, I was saying, everyone can propose a project. Can you tell us a little bit about how you organize, how you do your research?

Mm-hmm. I think that is very different from even in school, or even in

the earlier tech companies. It's pretty different. It's that what we make sure is that we have good foundation and good infrastructure so that anyone can play with the model and can think of what they can improve with the model.

And then after a model releases, when they are free, they can play with the model. They can think of their own evaluations. They can find their own weaknesses and propose the thing that they want to improve on the model.

And then other people who are interested in that would propose to join the project, and they will work on it for a couple of weeks, or even a couple of months. And when they work out, the final thing is shipped to our model.

We use that in our final training, and it's shipped out to the audience.

Interesting. So you can have people working for a really long time on projects. When you say a couple of months, it can be like really deep exploration of what's possible.

Yes. Yes. I would say, for example, architecture might require a longer time of investigation, research, experiments, even redoing the evaluations for pre-training. Yeah, so it might require a longer time.

That's really nice, yeah. And I know you're also very big on evaluation. I agree. We could talk about that. I think one thing probably related to that is this unique specificity that M3 and your team has around multimodality.

So not just text, but this model can also understand image and video. And as I understand, but please explain better, when we read the model card on Hugging Face, it said the model was trained from the first step as a multimodal, not just as a user one after sort,right?

Yes.

Can you tell us a little bit more about that, and why you think it's important, and why starting from the first step on multimodal training, and not just post-training?

So we call it native multimodality. And so it is somehow typical for model labs to train the multimodal, let's say, vision understanding capabilities after the text pre-training is done. They put adapters and then train that part. But what we found out was that that would actually harm the text performance.

And the vision understanding performance wouldn't converge that well, because the model is kind of converging towards the text understanding, and it's just not the most optimal. And also not the most scalable, if you think about it. We want to scale the data,right?

And also, we can also some labs train this capability from halfway through the pre-training, for example, continued pre-training. But what we found that this would be very recipe-sensitive. It is different for the recipe would be different for different architectures, different data mixtures, different learning rates.

It's hard to control, hard to scale to you can't really scale your experiment results and conclusions to a larger model. And so

what we thought was, why not just training from the very first step? That comes the most natural. We know that a lot of labs run into problems doing that. The model would collapse after a couple of steps of training both text and vision understanding.

But we managed to solve that problem. We did a lot of work on the ID, and we did a lot of work on the data that we actually training. For example, we do interleave the data, what we call interleave the data.

It's actually natural data, but we keep the images and videos in, instead of masking it out. And we do some pretty good cleaning and masking on the data, and we do very good reward modeling, so that we train it from the first step, and it scales up a lot.

Yeah, it does not collapse.

That's really impressive. Impressive. Should we expect a much larger model in the future? So this one is still fairly small,right? It's 428 billion parameters, 23 active billion. Well, do you think you will go past the trillion?

Definitely. Yeah, definitely in the future. There are many tasks that wouldn't be able to the model wouldn't be able to perform very good at with smaller parameters. We are definitely going more ambitious than this.

That's great. Looking forward. Another interesting thing I always find fascinating about MiniMax is how you also have this whole range of apps and products,right? So I remember already so MiniMax started to open-source things on the Hugging Face platform in January last year.

So that's 18 months ago. And we were chatting a little bit about the team to understand what you were doing. And I remember, so you were already having a huge usage on some of these apps. Can you tell us a little bit how this started,right?

So was it basically you had a lot of apps, and then you thought, we have all this data, why not train a model? And then they build up a research team. How is the story there?

Our story is modeled from the first day. So I believe that multimodality model, a model that can understand all visions and outputs all modalities, was the first thing that our CEO planned on the first day, even before the company even started.

So that was the dream of AGI. I think that was very, very early, even before ChatGPT came out.

Wow.

Yeah. And then apps were something that comes along. Because you have some model capabilities, you want people to experience it well. Not many people can use it with API,right? We can't expect everyone to experience it with API. So we need good user interaction, interfaces, good apps, good scenarios that people can experience the model with.

I think actually those apps covered more than 300 million people around 200 countries globally, and I think over a million companies as well.

Yeah. This was mind-blowing when I heard about the size. And we don't often realize the size of this type of usage already. And that kind of brings me to the question around open-source business model and all of that, which is always an existing question, which is,right now it's nice to open-source model, but you also need to have some revenue stream,right?

So I guess M3 is something you decided, for instance, to give for free, and I think it's great for the world. How do you see this? Do you also have some specific models you use for the app? Do you think about do you think in the future you'll keep it's probably hard to say for sure, but do you think you'll keep open-sourcing models?

How is the culture around open-sourcingright now?

Personally, and also for the model research team, we always hope to open-source the models. That is our plan, because we really see how the open-source community together can help the model build better. For example, we receive a lot of feedbacks on the model performance from the great community, and we receive PRs on whatever we open-source,right?

And those are very, very valuable when it comes to our later versions. So definitely, open-sourcing is great.

That's great. And actually, do you have some ask for the audience, people who are using M3 or MiniMax? Is there something you would love them to send back to you as feedback? Do you, for instance, do you read when people try to modify the models or play around tweaks?

Or what is the best thing you think you can take from the community for the future models, for instance?

I would say whatever issues that people are running into, especially with multimodality,right? This is the first time that we're combining it together. We are definitely going more ambitious on that in the future. It might have some flawsright now, but we are improving on that.

So whatever feedback that the model is not doing that great, we will definitely improve that in future versions. And also whatever features that people want. Say, for example, thinking effort,right? Some people ask for that. Everyone can ask, and we will try to accomplish that in the future models.

Yeah. Do you see a lot of usageright now already in multimodality, in terms of coding agents? I feel like it's a little bit underexplored.

It is. It is. But it can actually unlock a lot of capabilities and a lot of agent applications. Say that, for example, you want a model to read a PowerPoint, or to read some reports that is not very structured, and you want it to understand a very long video.

Say that you've done a long playing video, and then you want the model to act using some tools after understanding it. And it unlocks a wide variety of agent use cases.

So like the agent could finally watch my YouTube tutorial and understand how to use my coding tools, how I described it? Is it something like that?

Huh?

Could the agent finally watch YouTube tutorials and understand things from them?

Yeah, yeah, yeah. I think so.

Do you use a lot of agent coding tools internally? Is it like, I mean, coding for sure, but is it also already in terms of research? Is it automated in part or not? How does this roll?

Yes. We have our own research harnesses. We build our own research harnesses that automate our workflows. I would say a lot of our workflows are automated. You can see how the latest frontier models all pursue capability like kernel optimization,right?

Let the model pull string other models. Let the model build data, auto data, stuff like that. You can see how more and more models are capable of doing those, including M3. Actually, we were very good at those cases, longer horizons and kernel optimizations.

And so we can use that model capability, harness it together, and help with our daily routine, and make our iterations even faster.

Is M3 building M4 already?

Building M3.1.

M3.1, OK.

Yes.

That's yet to jump.

Already.

I would love to finish on what you find exciting in the coming month. What do you think? It can be Azure in terms of feature or things you want to see happening in AI, or more generally in terms of whatever really is top of your mind, I would say, that's going to happen.

A lot of things are very exciting. But what I recently find the most exciting would be multi-agents, that I think a lot of AI applications are using, model routing, multi-agents, that unlocks even more capabilities, even more complex tasks.

And also, it tells us what the models are capable and not capable of. And you can do a lot of things with that. It's pretty exciting.

Thanks a lot, Olive. Pleasure to have you.

Thanks for having me.

Thanks, everyone.