Perception Agents — Antje Barth, Amazon AGI Lab
Jul 23, 2026 · 21:45
Antje Barth of Amazon AGI Lab introduces perception agents — AI that sees and interacts with screens visually rather than through APIs — arguing that while agents can click and type, they fail at end-to-end knowledge work because reliability and trust are missing without verifiable outputs. Unlike coding agents, which succeeded due to unit tests, most work lives in the messy seams of applications where no easy verification exists. Barth's solution is a perception-action loop: the agent perceives the rendered screen, plans, acts, and checks its own work in real time, much like humans collaborating over a shared screen. She demonstrates two open-source tools: an annotation Chrome extension that lets users mark elements and issue precise commands, and a verification tool that checks visual design and user flows against specs. A live demo shows a Bee device capturing a meeting transcript that instantly triggers agent actions and verification. The talk closes with a call to build in the open, emphasizing that shared context between human and agent is the missing piece for reliable, trusted AI assistants.
