GEPA

Shipping AI to a Million Patients Without an A/B Test — Jared Joselowitz, Ufonia

Aug 19, 2026 · 19:15

Jared Joselowitz, research engineer at Ufonia, explains how the company ships Dora, a regulated medical-device voice agent, safely to patients without A/B tests, since randomizing patients into worse care is unethical. Dora has made 200,000 clinical calls across 20 UK hospitals and is contracted to reach a million patients in two years. Because 5% of patients is thousands and a red dashboard means someone was harmed, Ufonia built Matrix, where an LLM patient (PatBot) converses with Dora and a second LLM judge (BevJudge) flags hazards. A PPI study showed real patients found the simulated patient more realistic than a real patient in 3 of 4 sets, and the judge beat 10 clinicians on sensitivity. Prompts are optimized with GEPA against a cost matrix, because you ship the evidence, not the model.

Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops — Samuel Colvin, Pydantic

May 7, 2026 · 1:20:40

Samuel Colvin shows how to improve production agents without redeploying using Pydantic AI's GEPA, evals, and Logfire managed variables. GEPA uses a genetic algorithm to evolve prompts, achieving 96.7% accuracy — up from 87% with a simple prompt and 92% with an expert prompt — on a Wikipedia-based political relations task. Managed variables let you update prompts, models, and parameters live via a web interface and A/B test targeting. Evals compare against a golden dataset of 650 MPs; Colvin runs 65 cases in 30 seconds using GPT-4.1. He discusses that prompt optimization is most valuable with private data, that implicit user feedback (e.g., user's next action) can build golden datasets, and that overfitting to small test sets is a risk — the GEPA optimizer may exclude valid relations like 'uncle' if they don't appear in the training split.

Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI

Apr 10, 2026 · 40:51

Mahmoud Mabrouk, co-founder of Agenta AI, demonstrates how to build calibrated LLM-as-a-judge evaluators using the GEPA prompt optimization algorithm, arguing that miscalibrated evals are worse than none. He walks through a practical workflow for a customer support agent using the TaoBench airline dataset, covering metric design, data annotation, and GEPA-based optimization. The seed judge achieved 61% accuracy; after optimization, accuracy rose to 74% with reduced bias, though the judge still struggled to fully learn the complex policy. Mabrouk shares key lessons: start with a seed prompt biased toward compliance, use larger models for refinement, overfit to training data first, and beware of high token costs.

The Unreasonable Effectiveness of Prompt Learning – Aparna Dhinakaran, Arize

Dec 23, 2025 · 10:56

Aparna Dhinakaran, co-founder of Arize, argues that prompt learning—applying RL techniques to prompts rather than model weights—can continuously improve coding agents by auto-tuning system prompts from runtime feedback. She details a process using Claude Code and CLIMB on the SWE-bench dataset: agents generate patches, unit tests yield results, then an LLM-as-a-judge eval produces natural-language explanations of failures. These explanations feed a meta prompt that iterates on the system prompt rules. On 150 SWE-bench examples, this approach improved Claude Code's issue-resolution rate by 5% and CLIMB's by 15%. She contrasts their method with DSPy's GEPA optimizer, noting theirs required fewer loops due to carefully engineered eval prompts.