Browser Agents

From RL to IRL — Gaurav Mishra, Amazon AGI Lab

Aug 14, 2026 · 17:46

Gaurav Mishra of the Amazon AGI Lab explains why RL for computer-use agents works in games but breaks in real life, and how his team turns failures into training data. He shows early browser-training runs where an agent guesses its password and locks the account, and clicks a sponsored button styled like the submit button, landing elsewhere. The talk catalogs partial observability, irreversibility, expiring credentials, and ambiguous success, then proposes flight-school sandboxes, process reward models, calibrated confidence, and adversarial tasks. A later trajectory shows the agent recognizing the sponsored button, refusing to guess the password, and handing off to the user, echoing the talk's point that the difference between a demo and a product is what happens after the first failed click.

The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans — Corey Gallon, Rexmore

Aug 14, 2026 · 21:38

Corey Gallon (Rexmore) argues that an agent driving Chrome via the Chrome DevTools Protocol is just a 'meat bag with a mouse'—and details his open-source Chrome Agent tool. Preparing the talk earned him an OpenAI ban threat after his agent cleared Cloudflare Turnstile, MT Capture, Lemon, and reCAPTCHA v2 with no human in the loop. He argues for CLI over MCP: both succeeded ~83% in an Arise AI study, but CLI took 7 turns and under a minute versus MCP's 71 round trips and 8 minutes, and CLI can be 75x cheaper in tokens. The method is a sense/act/verify loop up a three-rung ladder: synthetic clicks, trusted CDP input, then human mouse paths with jitter, deliberate overshoot. For reCAPTCHA, deterministic code drives and rearms each round while the agent names grid tiles, as speed beats expiration clocks.

Bringing agents onto the world wide web — Paul Klein IV, Browserbase

Aug 14, 2026 · 18:26

Paul Klein IV, founder of Browserbase, argues browser agents are no longer held back by models; the missing piece is engineering. He details a three-part harness: multimodal agents that write code and intercept network requests, skills and memory that compress page context, and consistent infrastructure, mocking the 'SOC 2 compliant Mac Mini at scale' problem. Klein says the web itself must improve for agents, pointing to accessibility trees, Chrome's new Web MCP, and unsolved problems of agent login and trust certification. He contends the payoff is the logistics company in Singapore, the bank in South Africa, and the lumber factory in Mexico running on PHP forms, not San Francisco, and introduces Browserbase Agents as a battery-included solution.

Computer-use models will agentify the web, not APIs — Dhruv Batra, Yutori

Aug 14, 2026 · 21:00

Dhruv Batra of Yutori argues computer-use models, not APIs, will agentify the web's long tail. The head may expose endpoints, but 200 million active sites won't add MCP servers: restaurants post JPEG PDF menus and school districts answer procurement via FOIA requests scanned onto Google Drive. Reading HTML fails because scores and stock status load asynchronously and render as pixels, so the browser is a rendering engine: pixels are the source of truth, the bitter lesson for web agents. His Navigator model takes screenshots in, clicks out, writes JavaScript when faster, and verifies on screen; it hits 97% human eval on Mind2Web (8 of 300 wrong) at 80 cents per task versus $2.30. That endpoint will be another layer: simulated browsers clicking and returning structured results.

Browser Agents Don't Need Better Models. They Need Better Eyes. - Kushan Raj, ARK

Jun 28, 2026 · 4:26

Kushan Raj, a former Founding Engineer at Sarvam AI, argues that browser agents fail not because of weak models but because of poor interfaces: what the model sees, can do, and learns from. He built a browser-agent runtime that replaces raw DOM dumps (20k tokens) with a compact markdown representation (~1,800 tokens), uses stable action handles instead of one-click-per-call, and provides step-by-step feedback instead of pass/fail at the end. In demos, his agent using a cheap model completes multi-step tasks like downloading Aadhaar or booking a trekking site far faster than Claude, which gets stuck or scrolls unnecessarily. Raj plans to open-source the project and offer an API where users supply a URL and intent for execution, aiming to make browser agents faster, cheaper, and more reliable for everyone.

Building Agent Interfaces: Lessons from Chrome DevTools (MCP) for Agents — Michael Hablich, Google

Jun 5, 2026 · 22:38

Michael Hablich, product manager for Chrome DevTools at Google, presents four engineering lessons from building agent interfaces, centered on the Chrome DevTools MCP server. He introduces the metric 'tokens per successful outcome' to measure interface fuel efficiency, contrasting it with raw JSON trace files that blew through context windows. The team improved error messages, like adding 'Cannot navigate back, no previous page in history' to enable agent self-healing without human intervention. They decomposed a monolithic `debug_webpage` tool into 25 focused tools, but then faced a discoverability problem solved by auditing tool descriptions for intent. Hablich defends keeping autoconnect friction to prevent prompt injection, citing the 'lethal trifecta' of agents with tool access, data access, and internet access, and outlines three security tiers for browsing agents.

Introducing WebMCP: Agents in the Browser — RL Nabors

May 23, 2026 · 23:08

Rachel Nabors, former web standards contributor and now Principal Developer Experience Engineer at Arise, argues that chat-only agent interfaces (the 'starfish' design) are the CLI of the future and demonstrates how to replace them with rich interactive surfaces using MCP Apps and WebMCP. She explains STDIO vs HTTP transports, shows how MCP Apps bundle HTML, CSS, and JavaScript into single files rendered in agent iframes, and introduces WebMCP, which adds tool attributes to existing page elements so browser agents can call functions without screenshot parsing or DOM traversal. Using her own web comic site as a case study, she builds a comic reader with full panel navigation, transcript mode, and speech synthesis via the Web Speech API, and advocates for better client support of MCP resources to avoid inefficient tool calls for context loading.

Harnesses in AI: A Deep Dive — Tejas Kumar, IBM

May 17, 2026 · 20:27

Tejas Kumar, AI Developer Advocate at IBM, defines an AI harness as everything around the model that grounds it in reality, contrasting it with an agent loop. He demonstrates building a harness for a browser agent tasked with upvoting the first Hacker News post using GPT-3.5 Turbo, emphasizing that the prompt remains unchanged. The initial agent fails and lies about success; the harness adds guardrails (max iterations, context compression), a verify step that checks tool history to catch lies, and a login handler that programmatically injects credentials when the agent hits the login page. The final agent reliably logs in and upvotes. Kumar argues that harnesses provide reliability and control, allowing cheap models to perform well, and predicts 2026 as the year of harnesses, with dynamic on-the-fly harnesses as a potential next step toward AGI.

Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft

May 16, 2026 · 19:45

Marlene Mhangami of Microsoft and GitHub argues that AI boosts developer productivity only when paired with clean code practices, citing a Stanford study of 120,000 developers and GitHub Octoverse data showing 275 million weekly commits in 2026 with a growing share co-authored by AI. She advocates test-driven development (TDD) with Playwright for functional testing over unit tests, as AI-generated tests often affirm code behavior rather than user experience. In a live demo, she uses GitHub Copilot CLI and Playwright MCP server to automatically write failing Playwright tests for a toy store's search and filter features, then generate code to pass them. She emphasizes testing one feature per test, capturing screenshots for PRs, and committing code before fixes to preserve context. She closes with advice on using Playwright agents for complex state management and confirming cross-screen-size support.

Building Intelligent Research Agents with Manus - Ivan Leo, Manus AI (now Meta Superintelligence)

Dec 30, 2025 · 1:21:30

Ivan Leo of Manus AI introduces the Manus API and Manus 1.5, positioning the platform as a general action engine for building intelligent research agents. He demonstrates how the API enables asynchronous task dispatching, file uploads with automatic 48-hour deletion, and webhooks for scalable workflows. The workshop covers creating a Slack bot that integrates Manus's browser operator, file attachments, and connectors to private data sources like Notion. Leo shows real-time demos including a custom French learning app, a conference event scraper with calendar integration, and an invoice processing bot that references company policies. He also discusses the roadmap for memory persistence, document exporting (PPTX/PDF), and enhanced permission systems for browser automation.

From Arc to Dia: Lessons learned building AI Browsers – Samir Mody, The Browser Company of New York

Dec 19, 2025 · 17:48

Samir Mody, Head of AI Engineering at The Browser Company, explains how the team built Dia, an AI-native browser, after learning from their earlier browser Arc. He details three core lessons: optimizing tools for faster iteration by embedding prompt editors into the product itself; treating model behavior as a craft through a dedicated team formed after a strategy and ops employee rewrote all prompts in a weekend; and designing AI security as an emergent property, using confirmation steps for features like autofill to mitigate prompt injections despite the 'lethal trifecta' of private data, untrusted content, and external communication. The talk emphasizes that building an AI product required not just a technology shift, but a company-wide transformation in hiring, training, and collaboration.

Defying Gravity - Kevin Hou, Google DeepMind

Dec 2, 2025 · 25:10

Kevin Hou from Google DeepMind introduces Antigravity, a new agent-first IDE that combines three surfaces: an AI editor, an agent-controlled Chrome browser, and an Agent Manager. The central claim is that agents should live outside the IDE, enabling longer-running tasks and multimodal interactions. Key features include Artifacts—dynamic representations like plans, screen recordings, and diagrams—that replace raw chain-of-thought with visual summaries. The browser enables context retrieval and verification via screen recordings, while image generation allows iterative design through comments. Hou explains the research-product flywheel: DeepMind researchers use Antigravity internally to identify model gaps, improving capabilities like computer use and instruction following, which then ship in the product. He also details four categories of model improvements: intelligence, tools, long-running tasks, and multimodal, all powered by Gemini 3 Pro and related models.

Useful General Intelligence — Danielle Perszyk, Amazon AGI

Aug 2, 2025 · 19:58

Danielle Perszyk, a cognitive scientist at Amazon's AGI SF Lab, argues that general-purpose AI agents like Nova Act must be built to augment human intelligence rather than replace it, drawing parallels to how human cognition relies on controlled hallucinations and social co-evolution. She explains that current agents struggle with basic computer tasks like clicking, typing, and scrolling, so Nova Act combines a specialized version of Amazon Nova trained via RL for UI reliability with an SDK that allows developers to string together atomic actions (act calls) into complex workflows. The lab's vision centers on grounding agents in a shared environment and eventually giving them models of human minds, which requires a common language for humans and agents to reciprocally level up intelligence. Demos with teammates Carolyn (searching apartments) and Fjord (extracting data with Python and parallel browsers) show how Nova Act enables three-line code starts and granular control, positioning the product as a primitive for a cognitive technology that aligns representations and stabilizes human thinking.

Building Agents with Amazon Nova Act and MCP - Du'An Lightfoot, Amazon (Full Workshop)

Jun 21, 2025 · 1:26:20

Du'An Lightfoot and Benjiro Byami present a workshop on building AI agents with Amazon Nova Act, Model Context Protocol (MCP), and Strands Agents, demonstrating autonomous web navigation and multi-step task execution. Lightfoot explains agentic AI as plan, action, and reasoning, with components like LLM, knowledge base, guardrails, and tools. Nova Act, a research preview, enables browser automation: one demo searches Amazon for a coffee maker and retrieves the product title using natural language commands instead of explicit HTML tags. Byami builds an MCP server for Nova Act, allowing clients like Claude Desktop to control the browser via spoken instructions, then integrates it with Strands Agents, an open-source framework that combines a prompt, LLM, and tools in minimal code. The workshop shows parallel execution (e.g., comparing three monitors) and multi-agent collaboration for tasks like generating a PowerPoint presentation for cloud migration. Limitations include inability to bypass CAPTCHAs and occasional loops in complex websites.

The Web Browser Is All You Need - Paul Klein IV, Browserbase

Jun 20, 2025 · 17:31

Paul Klein IV, founder of Browserbase, argues that the web browser is the default MCP server for the rest of the internet, serving as the essential bridge between AI agents and legacy websites that lack APIs. He distinguishes between web agents (one prompt to many actions, e.g., OpenAI's Operator) and browser tools (one action per prompt), noting that both rely on vision-driven or text-based approaches to parse pages. Klein explains that Browserbase offers a horizontal MCP server for browsing, enabling automation of any website, and stresses the need for custom evals and observability to track agent behavior. In a live demo, he shows an agent navigating sfpca.org to find a dog for adoption, handling unexpected modals. He also addresses captchas, advising good citizenship and hinting at future agent authentication solutions.

Invisible Users, Invisible Interfaces: Accelerating Design Iteration with AI Simulation - Alex Liss

Jun 3, 2025 · 12:37

Alex Liss, VP of Data Science and AI at Huge, argues that AI should be used as a tool for design itself rather than as a feature, proposing Intelligent Twins—AI simulations of user personas—to accelerate needfinding and interface testing. Citing a trust gap where only 32% of US adults trust AI, he advocates for repairing it by designing better interfaces, not adding chatbots. He demonstrates a global audit of sports websites (NBA, Olympics, Premier League) where Intelligent Twins simulated casual and super fans across 72 tasks, revealing strong navigation but declining performance in information architecture and engagement. The methodology uses computer use agents for visual evaluation and human-in-the-loop review to generate focused design briefs. Limitations include reproducibility and the need for test-and-control validation, with future potential in integrating MCP protocol for turning Figma components into code. Liss concludes that AI simulation can help teams gather insights faster and create simpler, trustworthy interfaces.

The Current State of Browser Agents - Jerry Wu and Wyatt Marshall

Jun 3, 2025 · 21:13

Jerry Wu and Wyatt Marshall of Halluminate evaluate browser agents with WebBench, a benchmark of 5,000 tasks across 500+ websites. They find read tasks succeed at ~80%, but write tasks drop below 40% for autonomous agents, with failures often due to infrastructure like CAPTCHAs rather than agent intelligence. They advise AI engineers to choose read over write use cases, test multiple browser infrastructure providers, and combine agents with deterministic Playwright workflows for production. Notable emergent behaviors include an agent bypassing Cloudflare by searching Google and an agent posting a top-liked Medium comment.

How Deep Research Works - Mukund Sridhar & Aarush Selvan, Google DeepMind

Mar 26, 2025 · 15:15

Aarush Selvan and Mukund Sridhar from Google DeepMind explain how Gemini Deep Research works, a personal research agent that trades latency for comprehensiveness by taking up to five minutes to browse the web and synthesize reports. They discuss product challenges like building an asynchronous experience in a synchronous chatbot, setting user expectations, and presenting thousand-word outputs. Technical challenges covered include iterative planning with partial information, handling the fragmented web with entity resolution, managing growing context via recency-biased retrieval, and ensuring robustness to intermediate failures. The episode also explores future directions: going from aggregating information to providing expert-level insights, personalizing research to user roles, and combining web research with coding, data science, or video generation.

Navigating AI’s Frontier in 2025 - Grace Isford, Lux Capital

Mar 13, 2025 · 17:55

Grace Isford, partner at Lux Capital, argues that while 2025 is a 'perfect storm' for AI agents with reasoning models like O3 and R1, cheaper inference, and billions in infrastructure (e.g., Stargate, DeepSeek), agents still fail due to cumulative errors—decision, implementation, heuristic, and taste—exemplified by OpenAI Operator booking a flight incorrectly. She prescribes five strategies: curating proprietary and agent-generated data, building personalized evals for non-verifiable domains (e.g., seat preference), designing scaffolding that prevents cascading failures (citing Ramp's approach), treating UX as the moat (e.g., Codium, Harvey, TLDraw), and building multimodally with voice, smell via Osmo, and touch for embodiment. The talk, recorded at the AI Engineer Summit 2025 in NYC, closes with a call to reframe perfection through visionary product experiences.

This video was edited with AI agent. But how?

Feb 22, 2025 · 5:00

Muhteşem from Re-Skill presents the world's first open-source video editing agent, built in collaboration with Diffusion Studio, which uses LLM-generated code to edit videos via a browser-based UI. The agent starts a Playwright browser session and connects to operator.diffusion.studio, a web app built with Diffusion Studio Core that renders video directly in the browser using WebCodecs. Three tools drive the process: VideoEditingTool generates and runs JavaScript code based on user prompts, DocsSearchTool uses RAG to pull relevant information from llms.txt, and VisualFeedbackTool samples one frame per second and decides whether to proceed with rendering or refine further. File transfers between the Python agent and browser happen via Chrome DevTools Protocol, and for scalability, the agent can connect to a remote GPU-accelerated browser through WebSocket. The project emerged from Re-Skill's need for automatic video editing for personalized learning, where they found FFmpeg too limiting and turned to Diffusion Studio Core for its intuitive API and reliable in-browser rendering.

Beyond APIs: How AI Web Agents Are Automating the "Long Tail" of Knowledge Work

Feb 22, 2025 · 17:44

Arjun and Bhavani present Rtrvr.ai, a universal AI web agent using a text-based approach to autonomously perform tasks across multiple browser tabs at under a penny per page. They argue text-based reduces hallucination compared to vision-based agents like OpenAI's operator, and being a Chrome extension avoids password sharing and accesses logged-in content. Features include deep research navigating multiple pages, dynamic function calling for any third-party API, and graph generation. Rtrvr extracts structured data, performs actions like clicking dropdowns, and processes tabs simultaneously, automating the long tail of knowledge work including market research, LinkedIn automation, and WhatsApp messaging. They claim a distributed subtask approach lowers failure rates and envision collaborative dataset construction, e.g., aggregating local government events.

Which Jobs Can Be Replaced Today: Fryderyk Wiatrowski and Peter Albert

Feb 6, 2025 · 19:59

Fryderyk Wiatrowski and Peter Albert, co-founders of Zeta Labs, argue that autonomous browser agents will first replace reactive jobs—like customer support and scheduling—by automating low-leverage tasks while preserving human focus on high-leverage activities. They propose a trigger-pool system where agents react to emails, Slack, or events, requiring only approval for actions. Peter details building reliable agents: start with prompting optimized for model distribution, then add cognitive architectures (e.g., planning, scratchpads) to split tasks, and finally fine-tune with synthetic data or reinforcement learning. He advises minimizing noise in prompts, preferring text-based reasoning over images, and using language model judges to filter training data. The founders see continuous model improvement enabling agents to handle increasingly complex, proactive roles, moving toward full job replacement.

Emergence Launch: AI Agents and the future enterprise: Dr. Satya Nitta

Jul 31, 2024 · 5:41

Dr. Satya Nitta introduces Emergence, an R&D-led AI company spun out of IBM Research with talent from Google Brain, Alexa, and Meta, focused on advancing AI agents for enterprise. The company's core product, the Orchestrator Agent (G8, launching in August), plans, acts, and verifies across multiple LLMs and agents to automate complex workflows like claims processing. Emergence also develops Agent E, an open-source web agent that tops the Web Voyager benchmark by autonomously controlling the web. Nitta emphasizes the company's R&D on self-improvement agents, AI planning and reasoning, and agent-oriented programming, with applications in enterprise RPA and document processing. The goal is to enable enterprises to build and deploy autonomous agents that drive productivity by operating software and web browsers.