Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber
Jul 24, 2026 · 21:39
Soumya Gupta and Jai Chopra from Uber detail how they built closed-loop evals for their multimodal food photo enhancement agent, which edits images for Uber Eats merchants while preserving authenticity and avoiding homogenization. They describe a routing agent using a recall guardrail to decide whether to enhance or skip an image, and a pass at K metric for iterative enhancement with QA gates that check faithfulness, completeness, and realism. Examples include reward hacking where the agent overcorrected to a generic plate and failures like hallucinating extra chicken wings. They explain multiple feedback loops: a model loop for drift detection using human labels, internal dogfooding with thumbs up/down, and production metrics like conversion rates, all fed into a diagnoser that auto-tunes agents via a reflect-and-synthesize prompt optimizer, ensuring the system evolves without human intervention.
