# Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons — Kanish Manuja, Twilio AI Engineer · 2026-08-28 Kanish Manuja, principal engineer at Twilio, says an LLM gateway is a fight among availability, latency, guardrails, and cost, and degradation forces you to pick one. He prefers per-request fallback over retries and circuit breakers, with extra headroom for the backup provider; streaming commits you to provider A, so 'Something went wrong, please try again' is by design. Ignore gateway-wide latency—a reasoning model's normal is 2 to 60 seconds, a chat model's outage—and set timeouts per model per route. Guardrails fail too, so choose fail-open vs fail-closed, budget their time, and place them pre, parallel, or post; gateway dependencies need segregated keys and load shedding. Most teams want centralized governance, not a central gateway, so decentralize traffic and centralize governance. ## Questions this episode answers ### Why does the chat show 'Something went wrong, please try again' instead of switching to another model provider? Kanish Manuja explains the message is by design. Once a response starts streaming from one provider, tokens already sent to the client cannot be recalled, so mid-stream fallback is impossible. The generic error is the visible tradeoff of streaming's perceived speed, not laziness. [0:28](https://aiengineer.podhood.com/6c941169-67c2-424c-b8c0-2d2488ecb98b?t=28000) ### Why aren't retries and circuit breakers enough for LLM provider failures? Kanish Manuja says retrying an LLM API eats the latency budget very fast, and tripping a circuit breaker makes no sense when another healthy model provider is available to route to. Because LLM calls are slow and expensive, blind retries multiply cost and tail latencies; he recommends per-request fallback instead. [2:04](https://aiengineer.podhood.com/6c941169-67c2-424c-b8c0-2d2488ecb98b?t=124000) ### Why shouldn't you measure gateway-wide latency for an LLM gateway? Kanish Manuja calls a gateway-wide latency number a lie for mixed workloads, since embedding and classification requests may finish in under a second while chat takes seconds and reasoning models take much longer. He says to track P99 per model per route, and set timeouts per model class per route. [7:15](https://aiengineer.podhood.com/6c941169-67c2-424c-b8c0-2d2488ecb98b?t=435000) ### Should you build one central LLM gateway for your whole company? Kanish Manuja says a central gateway is a single point of failure, so he recommends rethinking it. In most cases teams actually want centralized governance, not centralized traffic. You can decentralize the gateway and still centralize governance through plugins, custom code, cost tracking, and rate limit management. [14:34](https://aiengineer.podhood.com/6c941169-67c2-424c-b8c0-2d2488ecb98b?t=874000) ## Key moments - **[0:00] Intro** - [0:13] Kanish Manuja: the 'Something went wrong, please try again' message is served by an LLM gateway that keeps working even when model providers are down. - **[1:09] Tradeoffs** - [1:09] Kanish Manuja: an LLM gateway is a fight between availability, latency, guardrails and cost, and during degradation you cannot maximize all four. - **[1:51] Availability** - [2:24] Kanish Manuja: retrying an LLM API eats your latency budget and blind retries multiply cost; use per-request fallback to a second provider. - **[3:01] Fallbacks** - **[4:35] Fallback traps** - [5:15] Kanish Manuja: streaming an LLM response commits you to the first provider, so 'Something went wrong, please try again' is a deliberate tradeoff, not laziness. - [6:13] Kanish Manuja: the fallback LLM provider should get even more capacity and headroom than the primary, because it is your last line of defense. - **[6:31] Latency** - [7:15] Kanish Manuja: a gateway-wide latency number is a lie for mixed workloads; track P99 per model per route. - [7:51] Kanish Manuja: 'A reasoning model's normal is actually a chat model's outage' — set per-route timeouts to avoid silent outages. - **[8:17] Reasoning** - [8:17] Kanish Manuja: reasoning and router models make latency unpredictable — Twilio saw P99 hit 60 seconds on the same prompt; set the reasoning level per route. - [9:24] Kanish Manuja: hedge the P99 tail by firing a duplicate LLM request when the primary request has consumed P90 of its latency budget. - **[9:40] Guardrails** - [10:12] Kanish Manuja: guardrails are just another unreliable service, so choose fail-open or fail-closed and default to the worst case you can live with. - [12:04] Kanish Manuja: run guardrails in parallel for structured outputs instead of streaming them; pre-hooks add serial latency and post-hooks are for auditing. - **[12:59] Gateway dependency** - **[14:34] Central gateway** - [14:34] Kanish Manuja: most teams that want a central LLM gateway actually want centralized governance; decentralize traffic and centralize governance with plugins. - **[15:44] Closing** - [15:44] Kanish Manuja closes by noting it is his son's birthday and asking the audience to prevent one incident for him and their customers. ## Speakers - **Kanish Manuja** (guest) ## Topics AI Infrastructure, Guardrails ## Mentioned Twilio (company), LLM gateway (product) ## Transcript ### Intro **Kanish Manuja** [0:13] I'm Kanish Manuja, I'm a principal engineer at Twilio. Let's start with a quick show of hands: who here has seen the message "Something went wrong, please try again"? Well, we have a few lucky ones, and a few that have had a good lunch. Um, so behind that simple message is actually a system that is very complex, that serves you that message despite the model providers being down. And that's what we're going to productionize today, or discuss productionizing today. So what is an LLM Gateway? An LLM Gateway is an entry point, or a middleware, between your apps and the model providers behind them. It does a bunch of things: routing, authentication, fallback, rate limits, all kinds of governance that you can think of. Andright at the heart of the gateway is a fight between four things: its availability, latency, your guardrails, and cost. In case of a degradation, you cannot maximize all four; you need to pick what you want. So with this talk, if you use an LLM Gateway, I want to hel- I want to help you to make that tradeoff for your use case. ### Tradeoffs **Kanish Manuja** [1:35] And if you design a gateway, I want you to design or provide those levers to your callers and customers, um, so that your customers are happy. Let's start with availability. If you have a single model provider, their ceiling is your ceiling; their outage is your outage. ### Availability **Kanish Manuja** [2:04] So in typical software engineering, the way you tackle unreliable dependency is by retrying. Retrying with exponential backoffs, with jitters, and when all of that fails, you have a circuit breaker that trips after you've seen sufficient failures and you stop calling the damn thing. This is not enough for LLMs. LLMs are very different compared to your fast, cheap APIs that you retry on. Retrying an LLM API eats into your latency budget really fast. And also, tripping over a circuit breaker when you have another perfectly fine model provider to route to doesn't make sense. You should use the second model provider. And third, as I said, the calls are slow and expensive, so blind retries just multiply your cost and your tail latencies. So what is a better idea here? It is actually a per request fallback. What that means is you can actually try model provider A and then, in sequence, try model provider B if your request to model provider A fails. ### Fallbacks **Kanish Manuja** [3:16] Another option to consider here is you can fire requests to both the providers in parallel, but that's only if you're highly, highly obsessed with latencies, because that's just going to double your cost. Some of the similar circuit breaking patterns apply here to LLMs as well. If you know that your primary has been failing for some time, it doesn't make sense to try it again. You put it you take it out of the load balancer or your request path and put it in a cooldown, and then after a few minutes have passed, try putting that back again. One interesting choice that you have to make here is where your failure counts live. You can decide to have the failure counts live in memory on the instances that are serving your traffic, or you can have shared infra where your failure counts are shared across the fleet. There are tradeoffs. If you want quick failovers, then fleet-wide helps, and with instance, um, with local state counters, the issue that you run into is whenever you change your deployment size, your configuration and your expectations change. So something to consider. ### Fallback traps **Kanish Manuja** [4:35] What that clean diagram did not really show you are some of the other gotchas that I'm going to discuss. So fallbacks are not transparent. While the industry is converging on an OpenAI API-compatible format, I would say there are still nuances, so you need to really test your fallbacks well. They can have differences in your tool calling schemas, token limits, stop reasons, and what have you. So with LLM gateways, you can have a normalization layer that can ensure that you can do cross-provider, um, fallbacks as well. Another thing is streaming. It so essentially, nobody wants to wait for 30 seconds to have a wall of text appear in front of them. So there are use cases where streaming is absolutely required, but it comes as at a cost. You trade away your levers. You cannot once you have decided to go with provider A, you have to continue going with provider A. You cannot mid-stream change the providers. Whatever has been sent to the client, it's done. And that's where the "Something, uh, went wrong" message, that's the one that you see. It's not because of laziness, it's by design, uh, that you see that, and it's one of the tradeoffs. I would like to call out one other thing where I've seen teams trip over and over again. They really provision and test their primary providers really well, but they the second provider, the fallback provider, doesn't necessarily get the same level of love. And I would argue that your throughputs, or your capacity, or your headroom should be even higher for the second provider or the fallback provider, because that's your last line of defense. If that goes down, your application goes down. Let's discuss latencies. Availability failures areright in your face. They fail. You get alarmed, you get paged. But high latencies can be the quiet ones, and they need to receive more love, um, than I would say tuning your services for just availability. ### Latency **Kanish Manuja** [6:54] One thing to call out: a gateway may run mixed workloads, and you can have embedding embed- embedding requests that take just less than a second. You can have classification requests that take less than a second. Uh, you have chat requests taking 3 seconds and reasoning requests taking a long time. Quick show of hands if you measure your aggregate latency for your entire service. Well, that was a trick question, sorry. You shouldn't. It doesn't make sense. It's a lie. You should be tracking your P99 per model per route, not a gateway-wide number. A gateway-wide number doesn't make sense, especially if you're running mixed workloads. And I hope you're not, um, for those who raised your hand. Another thing that can really I cannot emphasize this enough is for you to set timeouts on per model class per route. That's where that's the number one root cause of your silent outage. If you don't have a timeout, your gateway thinks your ha your request is being happily served, while it is not. And I'll leave you with this message for, for latencies, um, specifically. A reasoning model's normal is actually a chat model's outage, so you definitely need to track latency per route. ### Reasoning **Kanish Manuja** [8:17] Okay, this is the most painful or the, the slide that has given me the most scars, which is reasoning and router models. So this is where truly the latency is unpredictable. And reasoning models, they do not give you they're highly undeterministic, more deterministic, undeterministic than your normal models. You cannot set the temperature to 0 in many cases, and the same prompt can take somewhere from 2 seconds to 60 seconds. And we've seen that in production where P99 suddenly popped to 60 seconds for no good reason. So that's while there's no magical solution to it, I would recommend that you at least start with fixing the reasoning level per route. So with router models, they hide that abstraction behind you. Like, they pick which models to run, and I would highly recommend that you at least make as much, uh, you make requests as deterministic as possible with an undeterministic system. Another idea is hedging the tail. You can have a you can fire another request if your primary request actually consumed, let's say, P90 of your latency budget. This can hedge the ta this can really hedge the P99 tail, um, for, for your services. Allright, this is one of my favorite ones. Um, to keep your model secure, you need to have guardrails. And with that, guardrails are necessary for preventing your services from prompt injection attacks, keeping PII filters in place, having toxicity filters, keeping the LLMs to stop swearing at your customers, all those good things. ### Guardrails **Kanish Manuja** [10:12] But just like a model provider, there are tradeoffs too. Guardrails are just like another service that can go down, that can be unreliable. And that's where you need to choose: do you fail open, or do you fail closed? When I say fail open, you can still serve the request even if your guardrails are down. Fail closed, you block the request and say, "Hey, I'm not available." That's the tradeoff between availability and security to a certain extent. While there's no universal answer, it really depends on your use case. You can decide, like, for example, a toxicity filter, if it's not up and running, you can still serve that request. So the default choice should be the worst case that you can live with. There are a few things that you can actually do to improve the behavior of your systems in face of, uh, you know, guardrails being down and, and managing just unreliability of the guardrails themselves. So the first is time budget. Your request should never be bound by your guardrail timing. It should always be the LLM that is the rate-determining step. So make sure that you have timeouts in place and those guardrails run with a specific time budget. Another important thing is fallback. You've heard you probably know, and I've talked about it. We always discuss fallbacks with regards to model providers, but guardrails are critical services too, where you can consider fallbacks, have secondary providers, secondary checks, cache decisions, uh, to keep your service available when a guardrail provider is down. Another interesting choice that pops up with regards to guardrails is the placement of the guardrails. Typically, you can place the guardrail in three ways. You can have a pre-hook that runs where the guardrail actually runs on the input. You can and that's probably the safest, uh, but it does add serial latency, uh, to your requests. Another one is in parallel. This is one of my favorites, but just to call out, streaming wouldn't work well here with, with parallel. So if you're specially producing structured output, please don't stream them. Uh, try to save your latencies and run, run these guardrails concurrently for your structured outputs. Another one is post-hooks. These are best for, um, output monitoring, auditing your outputs, and, and so forth. ### Gateway dependency **Kanish Manuja** [12:59] So, so far, we've all I've discussed all the things that can go wrong with regards to our dependencies. We haven't discussed that we are actually adding another dependency in the request path itself, which is the central or which is the LLM gateway itself. There are a few things where we have been bitten by, uh, and we've learned some lessons that I want to share with you if you're working on an LLM gateway or using one. One is shared limits. Make sure that your API keys are segregated per route, per use case, to the gr most granular possible, uh, to the most granular thing that you can imagine. Um, having a noisy tenant can be one of the biggest problems here. Another thing is load shedding. This is a feature that you should, uh, as part of your runbooks, game days, uh, make sure that the gateway that you're using supports load shedding. Because when you have a retry storm, it becomes really hard to just scale out. You cannot simply scale out services that is under a retry storm. And all these web servers, they have an internal queue, and they're configurable. Make sure that they're bounded and they cannot request ac they cannot accept requests that are unbounded. And if you want to have some custom logic, you can even have traffic prioritization here as well to make sure under load your most important use cases get served well. ### Central gateway **Kanish Manuja** [14:34] Last thing that I wanted to discuss is the whole idea of a central gateway itself. It is a single point of failure. So if you're thinking of having a central gateway for your entire company for to LLMs, I would recommend rethink that and see what are the reasons that you want it. What I've noticed is that in most scenarios, it's not the central gateway that they want. They want centralized governance. And there is a path forward where you can actually decentralize the gateway and still centralize government governance. So do not try to centralize your traffic, but you can have plugins, you can have custom code that can centralize your governance. Uh, governance can be in the form of cost tracking, rate limit managing management, and there are other solutions possible. So explore those before you chart on having one central gateway for your entire company. It can be managed by a single team, but I wouldn't recommend deploying it as a single deployment for the entire company, even though it's distributed. With that said, I want to end this talk on a personal note. So it is my son's birthday today, and I'm here talking to strangers about circuit breaking. So the least you can do for me is please go and prevent one incident for me and for your customers. ### Closing **Kanish Manuja** [16:03] Thank you. If you have any questions, yeah. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.