Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub — Arek Borucki, Hugging Face

Good afternoon, everyone.

I have a question: how many of you knows Hugging Face?

Nice.

How many of you already use Hugging Face?

Amazing, almost everyone. But I think we still have opportunity to grow our usage. My name is Arek Borucki. I work as machine learning platform and database engineer at Hugging Face. Today I would like to walk you through how Hugging Face scaled infrastructure and how we ended up serving 3 million models to developers around the world.

I would like to share architecture decisions we made, challenges we faced, and lessons we learned while scaling one of the fastest-growing open-source AI communities in the world. I hope you will enjoy it, and let's get started.

Before I dive into technical details, let's talk about scale. Today, Hugging Face serves more than 14 million users, and this number is growing very fast, especially in the last couple of months. We host 3 million public models,

1 million datasets,

50,000 organizations, and not only hobbyists or scientists. More than 30% of Fortune 500 use Hugging Face as a part of AI workflows.

Just to give you some perspective, a few years ago we had 20,000 models. Today, 3 million. It is around a 150x increase in just the last couple of years. And this growth is exactly why I'm here today, talking about infrastructure decisions that keep the hub healthy at scale.

This is how fast the number of public models is growing on the hub. Every big release like LLaMA or DeepSeek generated thousands of new models on top,

and our infrastructure needs to handle that. And it is not only models. Also datasets. In 2022, we had 10k. In 2024, 100k. Less than a year ago, we had 500k. Today, 1 million. All this data must be stored, indexed, and also must be searchable.

And that's the hardest part.

And this is also the reason why we had to rethink our search. At 20,000 models, any query is fast, even without an index. Trust me, no one would notice. At 3 million, the same approach breaks. Imagine what would you do if the hub search would be slow.

You would just leave and go somewhere else. And this is also what users are doing. They expect fast, instant results. With 14 million users, even 1% is not a small number. It is 140,000 of people hitting slow search.

At scale, p99 is much more important than p50, and we are paying lots of attention to p99. And that's the reason why we invest in precompute tokens, denormalize, optimize for read collection in MongoDB, full text search based on Apache Lucene, Kubernetes autoscaling, and soon in database sharding.

The next slides will show you how.

High-level architecture. When users interact with the Hugging Face hub, his request flows from the front end to the hub API. The hub is running on Kubernetes. Currently, we are using horizontal pod autoscaler. During spikes, new pods scale up automatically to handle the load and scale back down when traffic drops.

This helps us to keep the hub healthy without manual intervention. Next, the request

goes to MongoDB Atlas, which is source of truth for our metadata. And there is one point that sometimes surprises people. MongoDB does not store the models themselves. It stores everything about the models. What does it mean in practice?

In MongoDB, we hold all the metadata: users, repositories, models, datasets, buckets, spaces, information, configuration data, billing data, access control, and more.

The actual models' artifacts, tokenizer files, cart assets, and configuration files are stored separately in cloud object storage such as AWS S3. This separation of concern lets us scale metadata independently from binary storage and compute independently from both. We can optimize each component individually for specific workload.

Now, let's check how search works in detail. For example, someone wants to search the model on the hub. And let's say that's LLaMA. So someone typed "llama" into Hugging Face's search bar. His request flows through the hub to an optimized read collection on MongoDB.

And this is not our main repo collection where we keep all the data. It's a separate denormalized copy only for reads and listings.

The key information is on the left. We tokenize model names on insert time, not at query time. For example, someone wants to publish model meta minus LLaMA slash LLaMA 3.18b. We split the long model name into small tokens like meta, LLaMA 3.18b, and we store them in an array in MongoDB document.

Next, Atlas search, which is using Apache Lucene under the hood, uses autocomplete to find matching models instantly.

This is an example of a single document from our model collection. In this example, I'm using findOne method. I want to find model ID meta minus LLaMA slash LLaMA 3.18b. So that's the model from the previous slide. And I'm projecting only search tokens array.

And we see that all those precompute tokens are part of this array. So we have meta, LLaMA 3.1, meta, LLaMA, etc.

Next, there must be a query. In the past, we were using classical MongoDB find methods on models collection with regex operator. And this regex operator, we're searching in search tokens arrays, models which are equal to LLaMA.

And then we were sorting results by trending score, which is calculated every five minutes. This is the number of downloads and number of likes, as far as I remember from the last seven days. This solution was working well as long as the dataset was small.

Regex doesn't scale well. So when our dataset started to grow very quickly, we started to have big problems with latency. So we decided to switch to Atlas search. That's a feature which is using Apache Lucene under the hood.

So MongoDB doesn't provide in core MongoDB server full text search. There is additional process, MongoT. This MongoT process is a wrapper around Apache Lucene.

For end users, this is transparent. You are just using unified MongoDB query API. And

when you

use aggregation pipeline together with dollar search operator, MongoDB will know that you would like to search Apache Lucene index. Obviously, you need to put the name of this index, which is in this scenario model search. Autocomplete

model equal to LLaMA, path search tokens. And we still sort results by trending score. And this solution is much more efficient and is, so far, scaled well. So we don't have any more latency issues in our search bar.

First two results returned by previous query. First, meta LLaMA has trending score 33. Second one, 14.

But Hugging Face hub is not only search. We have hundreds of different services in Hugging Face which are utilizing, which are using MongoDB. To handle a million of queries, we use seven node MongoDB clusters. With multiple machines, we can distribute queries across multiple nodes.

So no single node becomes a read bottleneck. This is how it works. Application talk to the MongoDB cluster. All inserts, deletions, or updates go to a single primary because only primary can handle them. However, we are distributing reads across multiple machines.

We also have one analytic hidden node. What does it mean? This node is invisible from application. MongoDB driver is not routing any queries to this hidden node. This node is still replicating data from primary, but is not interacting with production traffic.

We are connecting directly to this node, and we use him for any kind of

reporting traffic or any kind of really heavy queries. All secondaries continuously tail the oplock from primary, keeping cluster in sync.

Now, let's have a look at what actually is running on secondaries. First, all queries which doesn't require the latest data go to secondaries. Only queries that must have strong consistency stay on primary. And we are paying lots of attention to this.

We are paying lots of attention to the queries which must run on primary. Second, complex aggregations. Aggregations pipelines that scan large amount of data, sort, group, or transform the data should not go on primary. They are heavy. Secondaries are a better place for them.

Third, change streams. We react to data changes in real time for several reasons. For example, cache invalidation, sync to different data store technologies, like for example, AWS Redshift, or for event-driven workloads. Those kind of operations are also not very light, and they should stay impossible on secondaries.

Fourth, all ad hoc queries, reporting queries, maybe some experimental queries go to hidden MongoDB replica set member, which is isolated from production traffic. The pattern is simple. Primary should focus on what only primary can do. Anything else can be pushed to different machines.

However, with 14 million users, 3 million models, and our growth, soon single MongoDB replica set will not be enough. The next step is sharding. Sharding means scaling your database horizontally. Instead of putting full dataset on one replica set cluster, we are going to cut data into pieces and put each piece on separate shard.

Each shard will have his own replication, primary and secondary. So we will keep replication just multiplied. The key difference between replica set cluster and sharded cluster is replica set keep full dataset on each node. Sharded cluster keep only part of the data on each shard.

And then if you want to scale horizontally more, you are just adding more shards. And then MongoDB balancer will balance data across all those shards. There is also shard key, which must be selected. This is not trivial operation, but the stock is not about choosing shard key.

This way, we are going to scale everything: CPU, memory, storage, reads, and writes.

Now, let's have a look at what is going on the hub level. The hub is running on Kubernetes. Currently, we are using horizontal pod autoscaler. When CPU or memory threshold goes above target, Kubernetes adds new pod automatically to handle the spike and scale them back down when traffic drops.

Our deployment, hub deployment, can scale from 10 to 500 pods, depends on the traffic. This is how we keep the hub healthy without manual interventions and without infrastructure over-provisioning. So this is also a cost-effective solution. However,

what happens if horizontal pod autoscaler wants to add new pods, but Kubernetes does not have free nodes anymore? This is where the second layer comes in. We are using CastAI for Kubernetes node autoscaling. When pods are pending because there is no capacity and Kubernetes scheduler is not able to schedule them, CastAI is adding new nodes, and then scheduler is able to schedule those pods.

So we have two layers of scaling. First one is at deployment level. Second one is at infrastructure level via CastAI. But we are going to migrate horizontal pod autoscaler to KEDA, Kubernetes event-driven autoscaling. The difference,

HPA scale only based on CPU and memory. KEDA scale on real application metrics like request per second or event loop utilization. It means scaling is driven by actual workload, not by resource utilization only. For example, pod can have low CPU but high request queue.

KEDA can see it. HPA not.

The best part of this architecture, you never have to think about it. When you push the model, search the hub, or download the model, it just works. This is what scaling million models is really about, keeping the user experience simple no matter how complex it gets under the hood.

Thank you very much. It was a pleasure for me to be here today, and I wish a nice day for all of you.

Thank you.