# 200 Million Patient Interactions Later — Vivek Muppalla, Hippocratic AI AI Engineer · 2026-08-19 Vivek Muppalla, head of AI engineering at Hippocratic AI, argues clinically safe AI voice agents can end healthcare rationing, citing 200 million conversations and 60-plus health systems. He details Polaris, their architecture running 31 models per call—one central model plus 30 specialists for labs, medications, scheduling—in parallel after each checks if it needs to speak. Their decoder-only audio LLM adds context and domain knowledge for drug recognition; single-word answers get rescored because 'no' heard as 'now' is catastrophic. With 4-bit quantization, speculative decoding, and KV cache compression, latency savings are reinvested into intelligence. But at 10,000 calls a day, 99% accuracy means 100 wrong appointments, so they use 7,000 clinicians and 450 tests to catch a 1% failure rate. ## Questions this episode answers ### How many patient conversations has Hippocratic AI completed? Vivek Muppalla says Hippocratic AI is 'over 200 million conversations in at this point,' with 200 million clinical interactions, zero significant safety incidents, deployment in over 60-plus health systems, and an 8.5 on 10 patient satisfaction rating. [0:29](https://aiengineer.podhood.com/aacd5c33-f080-4532-bc4c-088bb7dd4e59?t=29000) ### Why does Hippocratic AI run 31 models at once instead of one? Vivek Muppalla says Polaris, their constellation architecture, runs 31 models at any given point of time for every conversation: one central model handling the conversation plus 30 specialist models covering labs, medications, and scheduling, all executed in parallel. Each specialist first quickly checks whether it needs to speak; a singular model would be one point of failure, unacceptable for a patient conversation. [8:30](https://aiengineer.podhood.com/aacd5c33-f080-4532-bc4c-088bb7dd4e59?t=510000) ### How does Hippocratic AI make inference faster without sacrificing quality? Vivek Muppalla says quality is their constraint and speed is the work. Inference optimizations include lossless 4-bit quantization (shrinking math from 16-bit to 4-bit), speculative decoding where a smaller model generates tokens ahead and the main model checks them, and a KV cache compression system achieving over 96% hit rate and making prefill 18 times faster. [15:17](https://aiengineer.podhood.com/aacd5c33-f080-4532-bc4c-088bb7dd4e59?t=917000) ### Why is 99% accuracy not good enough for Hippocratic AI's scheduling? Vivek Muppalla explains the scheduling math: with over 10,000 calls a day, a 1% error means 100 people a day get the wrong appointment type, so 99% is pretty bad. Catching that 1% error rate statistically takes about 450 tests to be 99% sure, and 1,900 tests to catch it 10 times. [16:34](https://aiengineer.podhood.com/aacd5c33-f080-4532-bc4c-088bb7dd4e59?t=994000) ## Key moments - **[0:00] Intro** - [0:13] Almost no one in Vivek Muppalla's audience has ever received a proactive healthcare call from their provider, his opening show of hands reveals. - [1:53] "Do no harm, patient first, access for all" is the oath every Hippocratic AI employee takes, says Vivek Muppalla. - [2:08] Hippocratic AI has completed 200 million clinical interactions with zero significant safety incidents across 60+ health systems, says Vivek Muppalla. - **[2:24] Product demo** - [2:41] Hippocratic AI's demo call opens with an AI care manager identifying itself as AI and asking about health monitoring devices. - [3:45] Hippocratic AI's ASR resolves a misspelled drug name like P-R-O-Z-I-N to Prazosin using a finite medication list, says Vivek Muppalla. - [4:35] Hippocratic AI's agent escalates in real time to a live nurse when the demo patient reports shortness of breath. - **[5:11] The tradeoff** - [5:11] Generic voice stacks fail for clinical calls: safe models take tens of seconds to respond, while fast models lack clinical accuracy, says Vivek Muppalla. - [7:26] Latency optimizations buy back time that Hippocratic AI reinvests into more intelligence, making latency and intelligence a compounding flywheel. - **[7:44] Polaris** - [8:30] Polaris, Hippocratic AI's constellation architecture, runs 31 models per conversation: one main model plus 30 specialists covering labs, medications, and scheduling. - **[9:20] Speech recognition** - [9:39] Most apparent AI reasoning failures are actually mishearings: Spanish 'sí' gets transcribed as 'C' and Arabic drug names hit 30% word error rate, says Vivek Muppalla. - [10:55] A conformer projector preserves prosody so Hippocratic AI's ASR hears the how as well as the what, says Vivek Muppalla. - [12:14] Hippocratic AI runs a second scoring pass for single-word patient answers because a misheard word can be catastrophic in a clinical call. - [12:52] Hippocratic AI's ASR cuts medical word error rate by over 50% versus off-the-shelf models and is 3x faster at p99 latency. - **[13:07] Constellation** - **[15:02] Inference stack** - [15:02] "Quality is our constraint and speed is the work," says Vivek Muppalla on Hippocratic AI's inference optimization. - **[16:15] Evals** - [16:34] A 1% failure rate at 10,000 calls a day sends 100 people to the wrong appointment, making 99% accuracy unacceptable, says Vivek Muppalla. - [17:24] Hippocratic AI uses 7,000 trained clinicians who have run close to 800,000 clinical conversations to evaluate its platform, says Vivek Muppalla. - **[17:54] Safety & empathy** - [18:09] Polaris reaches 99.89% no-harm accuracy, while humans score 81% on the same rubric, says Vivek Muppalla. - **[19:14] People** - [20:15] "We're told you gotta pick two of these options around quality, speed, and safety. We didn't, and we decided to go with all of them," says Vivek Muppalla. ## Speakers - **Vivek Muppalla** (guest) ## Topics Voice Agents, Inference Optimization, Guardrails ## Mentioned Hippocratic AI (company), Heart (product), Polaris (product), Whisper (product) ## Transcript ### Intro **Vivek Muppalla** [0:13] Hope everyone's having a great conference. Super excited to get started. Uh, how many of you all—just quick show of hands—have ever gotten a proactive healthcare call from your provider? Looks like no one. Me neither. Uh, I'm—Vivek—oh, a few hands over there. I'm Vivek, I run AI engineering at Hippocratic. We've built a product that calls patients and can have clinical conversations, and we're over 200 million conversations in at this point. Here's the reality of, like, healthcare: across all of human history, the entire system has been built on scarcity,right? Not enough clinicians, not enough time, not enough money, and hence the word "triage." We're always trying to figure out who amongst us is the sickest and deserves to get care and attention. As an engineer, I like to think about the math. And for most of human history, the math has just never been in our favor. Until now. We're finally at the point where we've seen enough of technology progress, and we also have AI systems that can do clinically safe conversations, and the cost for these conversations continues to drop. So the math now flips. We can stop rationing, and you don't have to have calls just for the sickest 5%, but you can call everyone. So that's the mission. We're trying to build clinically safe abundance for all, and it's not about replacing clinicians. In fact, what we're trying to do is help clinicians reach everyone that they couldn't do before because there was simply not enough of them to do this work. So how are we doing this? Um, our product and our company is predicated on three things. First, do no harm. Next, patient first, and access for all. Um, and here's a cool thing. Uh, this is plastered across our walls in our company, uh, but it's not just a poster. Uh, this is an oath every employee in the company takes. I did too on, on my first day. Uh, and we built our product, uh, based on this oath. Um, and this is working at scale. Uh, at this point, we've had 200 million clinical interactions. We've had zero significant safety incidents. ### Product demo **Vivek Muppalla** [2:24] We're deployed in over 60-plus health systems and have an 8.5 on 10 patient satisfaction rating. Now, uh, I could put an architecture diagram and explain how all of this works. And don't worry, we'll do that. But before we get there, uh, I want to show you the product in action. So we'll have one of our agents interact with the patient so that you get a feel for what we're up to. **Guest** [2:49] I'm your AI care manager. I'm calling to check in on how you're doing with your health monitoring devices. Do you have time to chat now? **Guest 2** [2:57] I, I need to know if I'm talking to a human being or a, a, some kind of computer. You sound like either one. **Guest** [3:06] I understand your concern, Ms. I'm an AI, which stands for artificial intelligence. **Guest 2** [3:12] Okay. **Guest** [3:13] How are you doing today? **Guest 2** [3:16] Well, I was tired earlier, but I'm somewhat revived. I'm incredibly amused. **Guest 3** [3:23] Delivering at this level of care required building clinical capabilities you won't find anywhere else. On the safety side, accurate drug name recognition. **Guest** [3:32] Could you tell me the name of the medication you're currently on? **Guest 2** [3:35] Convulsants. And there's one other, but I don't remember the name. **Guest** [3:39] Could you help me out by spelling the name of this medication? **Guest 2** [3:42] P-R-O-Z-I-N, I guess. **Guest** [3:45] It seems like you might be talking about Prazosin. Does that sound. **Guest 2** [3:49] Yeah. **Guest 3** [3:50] Vital sign clarification. **Guest 4** [3:51] Then I took it again. It was 121 over 45, 91. **Guest** [3:59] Just to confirm, your blood pressure is 121 over 45 and your heart rate is 91,right? **Guest 4** [4:05] Right. **Guest 3** [4:07] Med stoppage detection. **Guest** [4:08] What about the nifedipine tablet? Were you able to stop taking that one as instructed? **Guest 2** [4:14] Well, I took it this morning. Well, no, I took it last night because I didn't have the other prescription, and I don't think that would keep my blood pressure down. **Guest** [4:24] I understand your concern about your blood pressure, and it's great that you're thinking ahead. I would recommend that you do talk to your primary care doctor about the nifedipine and get their guidance on what to do. **Guest 3** [4:35] And real-time nurse escalation when clinically necessary. **Guest** [4:38] How are you feeling? **Guest 2** [4:40] Okay. Just a little tired and weak. **Guest** [4:43] It's quite common to feel this way after being in the hospital. Could you tell me more about it? **Guest 2** [4:49] My legs feel weak when I walk. **Guest** [4:51] Got it. Along with the weakness in your legs, have you noticed any chest discomfort or difficulty breathing? **Guest 2** [4:57] I just was, was having shortness of breath for a while. **Guest** [5:01] I'm so sorry to hear that you're experiencing shortness of breath. Given your symptoms, it would be best to have a nurse speak with you immediately. **Vivek Muppalla** [5:11] Yeah. So that's a sneak peek into our product. And so we'll get into why this is hard and why a generic stack doesn't work for us. Okay. So if I were to strip out all of the complexity around the audio and speaking back, it boils down to two things: intelligence and latency,right? ### The tradeoff **Vivek Muppalla** [5:28] On the top left, you see some of our ever-evolving smart, intelligent models. These models are fantastic at what they do. They're clinically safe. But unfortunately, many of these models take tens of seconds to respond, sometimes over a minute. And that's completely useless when we're trying to have a two-way conversation on a telephone. On the bottomright, you have models that are fantastic. They're really fast, but then they're just not at the same level of clinical accuracy for us to have a safe conversation. When we started building out this product, where we wanted to be was on the topright corner. And in order to accomplish this, what we had to do was build a vertically integrated stack ground up, bit by bit, optimizing every part of the sta-stack. When we started out building our own models, we also had several seconds of, like, latency. But hundreds of optimizations later, where we landed at was a product that is insanely fast, but also doesn't lose its intelligence. And we continue to benchmark all of this consistently. All of these results are on our website and more. Um, and there's three typical ways in which folks build, uh, voice systems. One, you could use an ensemble of models, or you use cascaded models, or you use speech-to-speech real-time models. Again, all of these models and architectures are great at different things. As an example, let me take two specific benchmarks amongst so many hundreds that are critical for our workflows. Uh, lab results check and IVR navigation. Now, these are things that you don't typically hear about or see in the most common benchmarks. And most of the generic models out there don't perform well at those. And we needed our models and our product to be over 99% accuracy on those specific benchmarks, amongst other things. Uh, and that's one of the key reasons why we continued on investing our own stack. And latency is key to this entire system. So every time we work on an optimization, we buy back some latency, and we just don't bank that latency. We use that extra gap now to back more intelligence into the overall system such that we can have a more reliable conversation with the patient. ### Polaris **Vivek Muppalla** [7:44] Um, and when we do that, we go back, work on more optimization, and that flywheel compounds. So what seems like a tug-of-war between latency and intelligence for us is a compounding flywheel. So what does this entire machine look like? Uh, this is Polaris. This is our constellation architecture. Left toright, you have the system that hears. The middle is the brain that reasons. Theright is the system that talks back. And all of this round trip needs to be really fast. So on the left-hand side, what you're seeing are a collection of models, uh, that are used for speech detection. So everything from bilingual switching to background noise detection, um, to contextual understanding of the conversation. The brain isn't a singular model. We, in fact, run 31 models at any given point of time for every conversation. So we have one central model that's handling, ha-handling the conversation, but we have other 30 specialist models, everything from labs to medications to scheduling, uh, that are feeding input into this model. And then finally, on the output side, we have a custom personality voice, HD quality, and clinical documentation engine that makes sure the patient gets back all of theright information, and all of the same information flows back into the health system. The reason we have the system is because we see a singular model being as, like, one point of failure, and that's just unacceptable for a patient conversation. Uh, this architecture gives us the redundancy and safety that we otherwise couldn't get to. And the patient never can tell the difference. It's pretty seamless from their perspective. Let me double-click into some of these systems for you. Uh, first we'll talk about our, uh, ASR system. So the real world, as all of us know, is fairly loud and noisy, uh, but most of the audio benchmarks get recorded in a quiet room. ### Speech recognition **Vivek Muppalla** [9:39] And something we learned the hard way was most of what looks like model reasoning failures end up actually being model mishearing things,right? So the Spanish C or yes gets transcribed as the alphabet C, or you could have an Arabic drug name that gets over 30% ina-inaccuracy in terms of, like, word error rate. So we had to build a system to co-to combat all of these challenges. A typical speech-to-text system just takes in the audio and then outputs some text. So ours is a decoder-only, uh, audio LLM system. So we take in the audio, but in addition to that, we're giving it two additional pieces of data. One is the context around the conversation thus far, and second, it's the domain knowledge around that entire conversation. And that is sort of, like, the trick that makes this work for us. So under the hood, what's happening? So we have first, we have an encoder, which is the year. Uh, so we took an open-source Whisper v3 large turbo model and fine-tuned that on millions of clinical conversations, uh, to get to the accuracy we need. Next, we have a con-conformer projector. What this does is takes the audio and compacts it and projects it into tokens that the language model would under-understand. But the key over here is it also maintains all the prosody. So all the pauses and the stresses are maintained. So the model hears not just the what, but also the how. Next, we're passing in the context around, let's say, the medications, the sp-spatients taking in, or the task at hand. Are we trying to fill in a form? Uh, what kind of, like, details that need to go in there and the policy. And how this helps is now when a patient mentions a medication name, we aren't guessing from, like, an infinite list of medications, but we have the chance to optimize around a finite list, and that helps getting the word error rate down. Next, we also have contextual biasing as a part of, like, the training. So during the training process, uh, we have millions of these synthetic patients with their addresses, phone numbers, uh, and other details, uh, that are fed into the training process. So as a concrete example, let's say someone mentions their address on a conversation as 1100 Gary Boulevard. Now, that specific utterance is ripe for phonetic garbage for most ASR systems. But given we're feeding that as additional context to the audio LLM, we have a high degree of accuracy and are able to get to theright transcription, which then feeds into the brain accurately. Further, what we see is for most clinical conversations, many of the patient responses are mono-mono words,right? Uh, and those often get transcribed incorrectly. A now becomes a no, or a five becomes a fine. And in a patient conversation, that's catastrophic. So what we do is a secondary round of sc-scoring, specifically when we see a patient has said only a singular word. And again, we use the context of the entire conversation to do this. So all of these, like, help us bring down the medical word accuracy rate by over, uh, 50%, um, from what we see as standard off-the-shelf models. In addition to that, uh, the specific system we've built allows us to, uh, check in the latency. And at a p99, it's 3x faster than every other system out there. Next, let's look into the constellation, uh, the brain. ### Constellation **Vivek Muppalla** [13:07] So as I said, we have 31 models live running in, uh, at any given point of time. So how do we do this without falling apart on latency? Uh, the answer is a bit counterintuitive. We actually run every single one of these models in parallel. Uh, but the trick over here is, uh, every single model does a really quick check, uh, to see if they even need to say something on this conversation,right? So every specialist first decides, "Hey, do I need to speak?" If not, it's a short circuit, and that's what helps us keep us in the budget. So that's the synchronous part. In addition to that, we also have asynchronous run models running in the background that are doing verification. And let's take two specific examples to walk through, uh, how we use these models. Again, going back to our medication example. For example, if the patient talks about ibuprofen or is talking about another medication, that's when the med engine specialist will kick in, and it will say, "Oh, I have something to say over here." Main model will pass in that context to the main model. The main model with along-along with everything else that already knows around the goal and the conversation will take this input from the specialist and steer the conversation as appropriate. Next, uh, like many other agentic systems, we heavily used, like, tool calls. And I think pretty much every one of us has experienced tool calls have, uh, innumerable failure modes. Um, so we have these, like, verifiers in the background that are running to make sure all the parameters into the tool calls and the responses and the tool calls are accurate. Uh, and for use cases like scheduling, this is what's gotten us to get to a 3.949 accuracy. And finally, for certain use cases, we're also running these verifiers offline where possible. Um, so again, going back to scheduling, sometimes we have the luxury of time to go back and check to make sure, uh, if there's an inaccurate appointment, we can course-correct that on the back end or call a patient back and apologize. ### Inference stack **Vivek Muppalla** [15:02] Um, all of these changes would be meaningless if we didn't optimize our, uh, inference stack. Uh, and we have a fantastic research and engineering team that lives and breathes on this problem. Uh, and for inference itself, quality is our constraint and speed is the work. So we can never compromise on the quality of the output. Every speed optimization has to be, uh, lossless. We've worked on, like, many, many of these, like, optimizations. Let me talk you through three specific ones, uh, that we found to be the most meaningful. Uh, one is 4-bit quantization. Uh, so basically what we've done is we've shrunk down the math for, uh, model computation to be from 16-bit to 4-bit, again, lossless. And this has, like, really, uh, helped us with overall latency. The next is speculative decoding. So we have a smaller model that's generating the tokens ahead of time, and then the main model just checks this in one go. And this also has greatly improved latency for us. Finally, a KV cache compression system. So, uh, as these conversations get, like, really long in length, uh, we've figured out a way to keep a large chunk of these conversations warm on cache, giving us an over 96% hit rate, and then also helps us with the prefill portion, uh, which goes 18 times faster. So this style of across-the-board-board optimizations is what's, uh, getting us ahead, uh, during the product development phase. Now, evals are obviously key, as we've seen across many of these, uh, talks. Um, so some quick math,right? Uh, for example, let's take the scheduling use case again. ### Evals **Vivek Muppalla** [16:34] Um, with over 10,000 calls a day, uh, a 1-1% failure rate, like, sounds allright. Most agentic systems would claim 80%, 90% accuracy, uh, and that's great for them. For us, even the 99% is pretty bad, uh, because 1% error means 100 people a day are gonna get the wrong appointment type. Uh, and it's physically bad because they'll show up on the wrong date, wrong time, but worse off, they could actually be missing a critical, uh, appointment, and that's unacceptable from our standpoint. Uh, so it's not just an annoyance,right? Um, and the challenge over here is in order to catch this, like, statistically, the math, like, is, like, incredibly challenging,right? You need about 450 tests to be 99% sure that you can catch this, like, 1% error rate, uh, and 1,900 tests to be able to see that you've caught it, like, 10 times. Um, so you can't purely rely on synthetic data from our experience to be able to get to this scale of, uh, accuracy. Uh, so we use a combination of human beings and synthetic data. So we have over 7,000 trained clinicians on our platform who are continuously helping us, like, evaluate our platform. Uh, they've done over, uh, 700,000, close to 800,000 clinical conversations to help us figure out the accuracy of our system. And we do this on a continuous basis. Finally, uh, coming, uh, to safety, um, uh, we've shipped five versions of our product, uh, thus far, and each one's gotten better, um, compared to the previous one. So we don't just grade the output of our models on the correctness or validity of it. ### Safety & empathy **Vivek Muppalla** [18:09] So we use the same scale that as they do for human beings. So they're graded across correctness, no harm, minor harm, severe harm, and death. And across five generations, where we're at with our Polaris system is a 99.89%, uh, accuracy, uh, with respect to, like, no harm. And humans on the same rubric are at about 81%. It's not because, like, we're terrible, but, uh, AI systems don't get tired, and unfortunately, we do. And we also don't have the luxury of having 30-plus supervisors helping us at any given point of time. And finally, safe is not enough,right? A key part of the product is the empathy. For human beings to open up to AI systems, we needed to make sure, uh, the voice and how we actually lead these conversations are empathetic. And, uh, as with most other things during our product journey, we found that there weren't actually accurate or good benchmarks to measure the empathy of these systems. So we actually built one, which is called Heart. Uh, the paper for this is public. ### People **Vivek Muppalla** [19:14] Uh, I encourage you guys to, like, read it. Uh, and, uh, we, we had to make sure the end system was not just safe, but also empathetic so that people would actually prefer it and use it. And finally, uh, all of this, like, comes down to the people. Uh, we wouldn't be here if we didn't invest in, uh, people from an AI-native perspective. Uh, everything in our company across the last couple of years, uh, has been transformed, uh, our hiring processes to how we use, like, tokens. We still have unlimited tokens for everyone at our company. We also have two great programs for anyone who comes into, uh, um, Hippocratic. For early career engineers, we have an agent deployment residency program where they're trained to write some of the best, like, healthcare safest agents. And then for experienced software engineers, we have an AI residency program where we help you start, like, training models, uh, and build the next great system along with us. So yeah, we're told you gotta pick, uh, two of these options around quality, speed, and safety. We didn't, and we decided to go with all of them. And we're hoping, uh, the rest of us would come and build with us. Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.