Evaling Video Slop — Maor Bril, Character.ai
Jul 25, 2026 · 23:13
Maor Bril of Character.ai argues that evaluating AI-generated video quality requires pairwise comparison rather than absolute scoring, because CLIP score misses temporal incoherence and LLM-as-a-judge is too slow and expensive. His team trained a small Qwen3-VL judge using Bradley-Terry loss on pairs of real and deliberately broken footage, catching drift early by running the judge as a regression gate in CI—every AgentX release clears an eval wall calibrated against human scores. The judge scores a 15-second video in three seconds, and they avoid becoming an AI detector by ensuring consistent encoding and annotation across real and AI footage. They also evaluate sound using Atmos and correlation with key frames, but admit lip syncing remains unsolved. The fix: score the axes you care about (story, pacing, physics) and put eval inside the generation loop.
