Shipping AI to a Million Patients Without an A/B Test — Jared Joselowitz, Ufonia
Aug 19, 2026 · 19:15
Jared Joselowitz, research engineer at Ufonia, explains how the company ships Dora, a regulated medical-device voice agent, safely to patients without A/B tests, since randomizing patients into worse care is unethical. Dora has made 200,000 clinical calls across 20 UK hospitals and is contracted to reach a million patients in two years. Because 5% of patients is thousands and a red dashboard means someone was harmed, Ufonia built Matrix, where an LLM patient (PatBot) converses with Dora and a second LLM judge (BevJudge) flags hazards. A PPI study showed real patients found the simulated patient more realistic than a real patient in 3 of 4 sets, and the judge beat 10 clinicians on sensitivity. Prompts are optimized with GEPA against a cost matrix, because you ship the evidence, not the model.
