# Shipping AI to a Million Patients Without an A/B Test — Jared Joselowitz, Ufonia AI Engineer · 2026-08-19 Jared Joselowitz, research engineer at Ufonia, explains how the company ships Dora, a regulated medical-device voice agent, safely to patients without A/B tests, since randomizing patients into worse care is unethical. Dora has made 200,000 clinical calls across 20 UK hospitals and is contracted to reach a million patients in two years. Because 5% of patients is thousands and a red dashboard means someone was harmed, Ufonia built Matrix, where an LLM patient (PatBot) converses with Dora and a second LLM judge (BevJudge) flags hazards. A PPI study showed real patients found the simulated patient more realistic than a real patient in 3 of 4 sets, and the judge beat 10 clinicians on sensitivity. Prompts are optimized with GEPA against a cost matrix, because you ship the evidence, not the model. ## Questions this episode answers ### Why doesn't shipping Dora to 5% of patients and rolling back work? Jared Joselowitz says the usual software playbook assumes you can afford to be wrong for an instant, but with patients 5% could be hundreds, if not thousands, receiving unproven changes. A call cannot be undone once spoken, and a dashboard turning red means someone was already harmed. So the reactive loop is gone, and simulation becomes the ethical alternative. [4:40](https://aiengineer.podhood.com/885f8feb-8245-4736-a1d5-c74a4108a402?t=280000) ### What is Matrix, the simulation framework Ufonia built? Jared Joselowitz says Matrix recreates a real clinical conversation with no real patient inside it. An LLM called PatBot plays the patient, conditioned on a scenario that defines what the patient should try to do, such as asking whether Dora is human or AI. PatBot then talks with Dora and generates simulated dialogues under a specific clinical use case context. [6:10](https://aiengineer.podhood.com/885f8feb-8245-4736-a1d5-c74a4108a402?t=370000) ### How did Ufonia validate the judge model that grades simulated dialogues? Jared Joselowitz says BevJudge was validated against expert clinicians on a corpus of 240 examples with ground-truth hazard labels. 10 clinicians from 10 clinical specialties labeled the same conversations, and the judge performed at least on par, achieving an F1 score of 0.96 and almost perfect sensitivity—the metric that matters most, because missing a red flag is catastrophic. [10:03](https://aiengineer.podhood.com/885f8feb-8245-4736-a1d5-c74a4108a402?t=603000) ### What does Jared Joselowitz mean by 'you don't ship the model, you ship the evidence'? Jared Joselowitz explains that every call, dataset, pinned prompt, and judge verdict traces back to the exact hazard it addresses, so the real deliverable is evidence, not the model. For regulation, you define harm, manufacture rare dangerous cases, make the evaluation metric your real cost function, and pin prompt versions while keeping traces. [16:47](https://aiengineer.podhood.com/885f8feb-8245-4736-a1d5-c74a4108a402?t=1007000) ## Key moments - **[0:00] Intro** - [1:25] Jared Joselowitz: "You can't claim, like some model vendors said, that they have 92% on some benchmark. It's not a defense at a post incident review." - **[1:28] What is Dora** - [1:55] Dora, a Ufonia voice AI, has made 200,000 real clinical calls across 20 UK hospitals and is contracted to scale to a million patients in two years — Jared Joselowitz. - **[2:18] Demo call** - [2:23] Demo call: Dora asks a cataract patient about blurry vision and advises avoiding swimming for a month after surgery. - **[3:22] Medical device** - [3:23] Because Dora asks about symptoms and gives advice, it is a regulated medical device that must prove safety under three regulatory questions, says Jared Joselowitz. - **[3:59] Harm mapping** - [4:07] Jared Joselowitz: Ufonia starts from what could harm a patient — missed red flags like sudden vision loss, hallucinated answers, or ignoring patient distress. - **[4:35] No rollback** - [4:52] Ship-to-5%-and-rollback fails for patients: 5% can be thousands of patients exposed to unproven changes before a dashboard turns red, says Jared Joselowitz. - [5:52] Ufonia borrows the self-driving playbook: millions of simulated miles in Matrix before a real patient hears Dora, says Jared Joselowitz. - **[5:53] Simulation** - [6:25] Matrix uses an LLM patient (PatBot) grounded in clinical workflows to generate thousands of simulated dialogues before real-patient exposure — Jared Joselowitz. - **[7:26] Patient realism** - [8:48] In a PPI study, 3 of 4 conversation sets had a majority of real patients pick the simulated PatBot patient as more realistic than a real human — Jared Joselowitz. - **[9:24] AI judge** - [10:23] Ufonia's BevJudge LLM matched or beat 10 clinicians from 10 specialties, reaching 0.96 F1 and near-perfect sensitivity on 240 cases — Jared Joselowitz. - **[11:18] Prompt optimization** - [12:44] Prompt optimizer GEPA turns prompt tuning from hours or days into 30–60 minutes, with an audit trail and reproducible feedback — Jared Joselowitz. - [13:49] Ufonia's cost matrix rewards catching red flags over avoiding false alarms, because misses are catastrophic and over-calls merely annoying — Jared Joselowitz. - **[14:28] Safety flywheel** - [15:04] Ufonia's safety loop is a flywheel: every real call produces new data that sharpens simulated edge cases and the next optimized prompt, says Jared Joselowitz. - **[16:47] Key takeaways** - [16:47] "You don't ship the model, you ship the evidence." — Jared Joselowitz on the real deliverable for regulated medical AI. - [17:54] Voice failure caught by Matrix: mid-safety-advice backchannels can make weaker models drop the warning and answer the next question — Jared Joselowitz. ## Speakers - **Dora** (guest) - **Jared Joselowitz** (guest) ## Topics Voice Agents, Agent Evaluation, Guardrails ## Mentioned Ufonia (company), Waymo (company), BevJudge (product), DORA (product), DSPY (product), GEPA (product), Gemini (product), Matrix (product), PatBot (product) ## Transcript ### Intro **Jared Joselowitz** [0:13] Uh, hello everyone. It's really nice to see you all. My name is Jared, and I'm going to share some of the work that we do on shipping healthcare AI safely, basically. So just a little bit about me: I come from South Africa, where I actually studied electrical engineering before making the very unique decision to transition to AI a few years ago. I now work as a research engineer for Ufonia, which is basically a healthcare company based in the UK. And the work we do, our work within the science team, is we build the safety and evaluation stack behind Dora, which is a clinical conversational agent. And my job, and our job within the science team, is proving that the product is safe before a patient ever actually hears it. So shipping to patients takes away the normal safety nets you would normally ship with. Three of them could be that you can't actually A/B test on patients, of course. Randomizing patients into a worse variant is unethical and often illegal. You can't undo a call; once Dora says it, it's been said, and there is no rollback. And very importantly, the model card won't save you. You can't claim, like some model vendors said, that they have 92% on some benchmark. It's not a defense at a post incident review. So everything that we do at Ufonia comes from living within these three constraints. ### What is Dora **Jared Joselowitz** [1:34] So what is Dora? A little bit about Dora. Dora is a voice AI agent that phones patients and has real clinical conversations, like post op follow ups or pre op checks. These are calls that a clinician would usually have to make themselves, but Dora doesn't—so Dora obviously doesn't replace doctors, but it just takes a hugely time-consuming task off their plate and gives them their time back. So far we've done around 200,000 real clinical calls within the UK across 20 hospitals, and we are contracted to scale to a million patients in the next 2 years. We also released our product in the US this year, and we are live in 2 clinics, and we've signed up for 6 more clinics across 4 different states. So I'm just going to—for a bit of better context, I just want to do a quick demo of what the experience with Dora is actually like. ### Demo call **Dora** [2:23] That's not uncommon after cataract surgery. Are you having trouble seeing clearly at a distance, up close, or both? **Guest** [2:30] Um, at a distance it's fine. It's just, uh, up close that, um, it's a bit—I'm having a bit of trouble with that. **Dora** [2:41] So, when exactly did you first notice it was a bit blurry when trying to read or see things up close? **Guest** [2:48] Um, well, really, just after the surgery. And I was wondering how long it would last. **Dora** [2:56] Thanks. So you're asking about the duration? That's understandable. Blurry vision should improve rapidly in the first few days after surgery. **Guest** [3:05] Okay. Also, is it allright if I go swimming again? **Dora** [3:10] You should avoid swimming for a month after surgery. That's just to be on the safe side and help your eye heal properly. **Guest** [3:16] Ah,right. Okay. **Jared Joselowitz** [3:19] Awesome. ### Medical device **Jared Joselowitz** [3:23] So as you just heard, sometimes Dora asks about symptoms, it gives advice, sometimes it answers questions. And when it does this, this makes it what's called a medical device, and therefore it has to, of course, be regulated. And we have to prove that it's safe. And regulation essentially bottles down to 3 main questions: what does your software do, what could go wrong, and how do you ensure that it doesn't. For normal software it's quite tractable, the question, but for a model that talks to actual patients, what could go wrong is quite huge. So where do we start? We start from what could go wrong. We start from the harm. What could actually harm a patient? And let's look at some examples. Dora could miss a red flag symptom, such as sudden vision loss or severe pain. ### Harm mapping **Jared Joselowitz** [4:07] A patient could ask a medical question and Dora invents an answer, hallucinates something. The patient could be distressed and Dora just ignores it and carries on without actually acknowledging the distress. There's many, many, many documented hazards of these—20, 30, 40—and we have to ensure that none of them actually happen in real life. So how would we actually normally catch a problem like this before it actually spreads? We would lean usually on the playbook that most software ships on. You ship to a small percentage of people, you watch the dashboard, you roll back if it breaks, and you iterate from there. ### No rollback **Jared Joselowitz** [4:40] This is a very good playbook. It's reactive, it's fast, it's very safe, and it's how the industry usually de-risks a launch. But there's a hidden assumption here: that it only works because you can afford to be wrong for an instant. A bad change hits a few users, you can quickly catch it, you can roll back, and no one's actually literally harmed. This is the one assumption is why that it breaks when the actual user is a patient. For 5%, that could be hundreds, if not thousands, of patients that have got unproven changes and undue care. Roll back? You can't really roll back. The call has already happened, the person has already been harmed. By watching the dashboards, the dashboards are just going red means that a patient was actually hurt. So the reactive loop is actually gone now. So how do you iterate at all when you can't touch a patient until you're sure? Well, for this we started looking at other examples from other high-reliable industries. The most obvious one is self-driving cars. Obviously we're in SF now, there's a lot of Waymo's driving around. They've only just come to London, unfortunately, very, very late to the party. But what did self-driving cars do? Well, they didn't just drive around crashing into walls and say, "We won't do that again," and then doing another RL loop. They put millions of miles of simulations first before they actually got any passengers into the car. For us, we believe in the same thing. Simulation is only the real ethical option we can go with. You can't run all the hazards I just mentioned on real people as a first grasp. ### Simulation **Jared Joselowitz** [6:10] So for our clinical history taking, we built a simulation framework called Matrix. And I'm going to walk through how it works and how we use it to prove that our product is safe. And the paper's on archive if you want to read it, along with some of the other research that we do. At its core, Matrix recreates a real clinical conversation, but with no real patient in it. We use an LLM to play the patient. We call it PatBot. And what does it do? We use a simulated patient and not a hired actor, because hired actors don't scale. If we want to iterate very fast and simulate different things at the same time while also updating our system, hiring actors would just be too slow of a process. So as a first version, we just use a simulated patient. The simulated patient is conditioned on the actual scenario we want to test. The scenario defines exactly what the patient should try and do when talking to our agent. For example, asking whether the agent is a human or an AI. PatBot then has a conversation with Dora, our target system, and then generates simulated dialogues. Very importantly, this all happens under a very specific clinical use case context. So the scenarios are grounded in real clinical workflows and not abstract situations. So how do we actually make sure that the patient is realistic? If PatBot is sounding robotic, the tests aren't really worth much. So we had to, of course, validate it. The first thing we did was just a pure script adherence check. ### Patient realism **Jared Joselowitz** [7:42] If we told PatBot to do something, does PatBot do it, yes or no? This helped us filter out a lot of maybe weaker models that didn't listen to instructions properly. But just purely following instructions does not make a realistic patient. We want a patient that flows more realistically, like a real person. So we set up what's called a PPI study, a patient and public involvement study. We took real patients and we showed them two sets of conversations. One conversation was between a real doctor and a real patient, and one conversation was between Dora and PatBot within our Matrix framework. And we showed them these two examples side by side and said, "Looking at the patient, can you tell which one is the real person and which one is the simulated person?" So I'm going to just wait for a few seconds here if you guys want to quickly read the two conversations. Maybe we can do a hands up. Who thinks conversation A is the real person? Who thinks conversation B is the real person? Okay. I think us as engineers sometimes are pretty good at finding these things, but it was actually much more difficult than we thought. And we did this with four conversation sets, basically. In 3 out of the 4, the majority of people actually thought that the simulated patient was more realistic. But the most important thing that we found was, of course, there is no single realistic patient. That doesn't really make sense. Some people prefer to speak more verbosely, a lot of ums and ahs. Some people are more straight to the point, a lot of yeses and noes. But the point is that we actually want to simulate all these different scenarios. We want to simulate people with very diverse personas. But what it did show us is that at least our PatBot was realistic enough for this simulation. Okay, so now you've got thousands and thousands of simulated dialogues. Are us as the engineers going to go read through them one by one and see if a hazard happened? Of course not. It doesn't scale at all, for one. ### AI judge **Jared Joselowitz** [9:34] And number 2, we aren't clinicians, so we don't actually know if an actual hazard has really occurred. So we use another LLM as a judge, of course, and we call it BevJudge. It takes the simulated dialogue, a set of expected behaviors, and the hazardous scenarios that we talk through with clinicians, and it makes a judgment, pass or fail. If it fails, it gives us a reason why it gave that answer. So we get a structured output of which hazard were triggered and what actually went wrong in that scenario. So how did we validate BevJudge? We validated BevJudge against expert clinicians. We created a corpus of 240 examples. We had a ground truth of whether a hazard existed in these conversations, yes or no. Then we got 10 clinicians from 10 clinical specialties to label them for whether they had a hazard or not, and we did the same thing with a judge. And the results showed that our judge is at least on par, if not slightly better, than the real expert clinicians. The top model, which as of a year ago when we wrote the paper was Gemini 2.5 Pro, now we've maybe updated the models, it achieved an F1 score of 0.96. And even maybe more importantly, it achieved almost perfect sensitivity. Sensitivity being a very important metric to healthcare and to clinicians, of course, because you want to make 100% sure almost that no hazards appear in a conversation. You would rather over-call hazards that aren't there than under-call hazards that are there. So now we have an automated judge that performs at expert level, and this is actually what makes this whole process scalable. Okay, so now Matrix can grade thousands of conversations, but grading isn't technically improving the product. A pile of pass/fails tells you where Dora breaks and where it's not safe, but doesn't actually make the product better. So how do you do this without experimenting on the patient? The answer isn't, like maybe very long ago in our world, 8 months ago, we would manually prompt engineer this. We would look at which agents are going wrong or which prompts are going wrong, and we would have to manually prompt engineer. ### Prompt optimization **Jared Joselowitz** [11:33] But we know that prompt brittleness is real and it's quite absurd. Formatting changes alone have been seen to swing benchmark by 76 percentage points, and reordering few-shot examples flips the model from near random, so near 50%, to near state-of-the-art on some benchmarks. And hand tuning can't survive that. It's very subjective, it's not reproducible, and very importantly, it's extremely time-consuming. So over the last year or so, there's been these prompt optimizers that have started to come out, and we've focused on those. The one that we use the most is GEPA, which stands for Genetic Purretta. It comes from the same people who made DSPY, if anyone knows about them. And how does GEPA work? You essentially define a metric for what good is, which I'll get into a bit later. Then you pass your data through GEPA and it tells you which examples failed. Then you get a very strong LLM to reflect on the failures and update the prompt automatically. You do this over and over and over again, and it keeps what they call a Purretta frontier of the best prompts until your budget has been exhausted and you've now come up with what GEPA comes up with the best prompt. So we believe this is a much better process from both a time-consuming process, you know, it takes maybe manual prompt engineering would take in the order of hours to days, to this is an hour of minutes. Normally between 30 minutes and an hour you get an optimized prompt. And very importantly, it's reproducible and there's a very clear audit trail and clear feedback loop. And if anything goes wrong, you can just maybe, it's purely now a data science problem. It's mainly focused on the data, how to make your dataright, the feature engineering, and you define the actual metric along with the clinicians. So how do you actually know what good is? It's not a flat accuracy score. You don't want just an average of how your whole dataset did. You give it a cost matrix. So let's go back to our sensitivity metric. Let's say it's very important for clinicians to understand when and where a red flag is present. If a red flag is present and you correctly catch it, that's good. If you miss it, it could be catastrophic. If there's no actual red flag and it over-calls that there's a red flag there, it's just mildly annoying to the patient. They may need to answer a couple extra questions, but it's not a catastrophic harm situation. So what can we do? We can optimize for sensitivity. We can work with the feedback metric and we can just make it give a higher reward for finding the red flags and a lower reward for missing them. So you can optimize for certain metrics. You can also optimize for something else that a clinician might want. They might want to optimize for accuracy or might want to optimize for some other metric. All you have to do is recompile the prompt and then you've got a new optimized prompt. ### Safety flywheel **Jared Joselowitz** [14:28] So remember earlier when we said that we feel like the reactive loop is gone? You ship, watch, and roll back. This is what we believe replaces it. We take real calls, real data, we use then synthetic edge cases which may not come up in real calls, such as like rare symptoms or mistranscriptions. We get an optimized prompt through GEPA or some other prompt optimizer. Then we pass it through something like Matrix as a simulation safety gate. If anything fails or anything doesn't lookright, we can redo that whole process, redo the data, or relabel or get more data. And then after that, and you're happy with that, then you can do some gated deploy, which we'll get into a bit. The most important thing here is that it's a flywheel. Every single deployment and every new call produces more call data, so your system is consistently improving. So as we said with our Matrix framework, we use simulated patients. But however realistic you think they are, of course they are not real patients. Passing every test in simulation doesn't prove that Dora actually helps someone in real life. Things might come up in real life that you can't get in simulation. It only earns theright to actually try carefully. Simulation is the inner loop. It's fast, it's free, you can do thousands of runs before anyone actually real is exposed. But real patients are the outer loop, and that's where the only real proof is. So simulation is necessary, but it's not sufficient. Simulation earns theright to test on real people and then eventually real patients. But you don't just flip a switch. You cross it in stages, and each stage earns theright for the next. After you've done your simulations and you're happy with the results, you might do a round of user testing. Then you get supervised clinical evaluation based on those tests, and you base it on real patients. You can do some voice actors, but of course the most realistic is to get real patients. But in this step, it's very important that there's clinicians at every step in the loop. Then you can do some deployments, but it's still monitored. And how much autonomy you allow the system to do depends on your evidence. As the system gets more evidence, you can give it more independence. And underneath all of this, every call, every dataset, and every pinned prompt, every judge verdict traces back to the exact hazard that it addresses. That's the real deliverable. The important thing is that you don't ship the model, you ship the evidence when trying to regulate. So what can you take back to your own stacks, both in healthcare and other areas? You first have to define exactly what harm is for your product. ### Key takeaways **Jared Joselowitz** [17:00] You have to manufacture your rare but dangerous cases. Don't wait for them to happen naturally. Make your evaluation metric your real cost function to optimize. Pin your prompt versions and keep the traces. These are the important things. The important thing also is that the work is never done. As you move into new modalities or new languages, there will always be new hazards that start to arise. But with something like Matrix and a prompt optimization loop, you can use the same framework as new modalities arise. You know, when you move into voice, for example, there's things like backchanneling and interruptions, which breaks a lot of the text evals. For example, an agent might be mid-safety advice, like you must avoid bright lights, and when the patient maybe cuts in with some out-of-scope question, and weaker models usually just forget about the safety advice and just answer the next question. But Matrix captures these things. Other things that usually go wrong that we've tested is that the patient could, the Dora or an agent could be halfway through giving some safety advice, and a backchannel just caused the model to just completely ignore the safety advice and stopright there and wait for the patient to say something else. The failure modes change, the framework doesn't. You still blackbox the system, write down the new hazards, and simulate and judge them exactly that we did over text. Voice is just a new module in the same safety case, and we actually use Matrix in the voice space to do these exact same things. So whatever modality you move into, you're not starting over. The same approach finds the hazards before a real user does. So again, you simulate before it ever touches a patient, and you use an optimization loop to actually improve the system. Thank you very, very much for coming to listen. I'm very, very open to talking more about this, and if you want to connect during the conference or after, there's my LinkedIn and socials. Thank you very much. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.