Playwright

From RL to IRL — Gaurav Mishra, Amazon AGI Lab

Aug 14, 2026 · 17:46

Gaurav Mishra of the Amazon AGI Lab explains why RL for computer-use agents works in games but breaks in real life, and how his team turns failures into training data. He shows early browser-training runs where an agent guesses its password and locks the account, and clicks a sponsored button styled like the submit button, landing elsewhere. The talk catalogs partial observability, irreversibility, expiring credentials, and ambiguous success, then proposes flight-school sandboxes, process reward models, calibrated confidence, and adversarial tasks. A later trajectory shows the agent recognizing the sponsored button, refusing to guess the password, and handing off to the user, echoing the talk's point that the difference between a demo and a product is what happens after the first failed click.

Codex, Behind the Harness — Dominik Kundel, OpenAI

Aug 10, 2026 · 20:55

Dominik Kundel, an OpenAI engineer, explains the internals of the Codex agent harness, which is open source under Apache 2 and written in Rust. He details how context construction caps available skills at 2% of the context window and uses deferred tools with tool search to manage size and cost. For actions, Codex relies on an apply patch tool for file edits, a shell tool with ripgrep, and sandboxes: seatbelt on macOS, bubblewrap on Linux, and a custom open-source Windows sandbox. To reduce approval fatigue, an auto-review subagent with read-only permissions judges actions against user authorization and risk taxonomies. Speed improvements come from websocket mode in the responses API, which sends only changed items instead of full state, crucial when GPT 5.3 Codex Spark hit 1,000 tokens per second on Cerebras. Long-horizon goals work via a continuation prompt until the model calls an update goal tool, favoring concrete verifiable objectives, and auto compaction maintains performance…

Full Workshop: Setting Yourself Up for Success —Jason Liu, OpenAI Codex

Jul 24, 2026 · 1:15:02

Jason Liu walks through how to set up OpenAI's Codex for maximum productivity, using voice dictation, app shots, computer use, and a personal memory vault to delegate and automate almost every task. He demonstrates creating skills and plugins from past work, using compaction to keep threads running for weeks with hundreds of sub-agents, and setting up automations with heartbeats and goals. Specific examples include auto-checking flights, editing iMovies via computer use, and having a chief-of-staff thread that monitors Slack and updates task lists. He also covers security permissions (auto-review vs full auto), tips for new users to start with low thinking mode to save tokens, and how to build skills that self-improve by editing their own files. The workshop emphasizes that threads can now communicate with each other, enabling manager-like orchestration and long-running work streams.

The Future Is Domain-Specific Agents - Justin Schroeder, StandardAgents

Jun 29, 2026 · 30:38

Justin Schroeder of StandardAgents argues that domain-specific agents—small, focused AI agents each limited to a single domain—will outperform general-purpose agents that rely on accumulating tools and context via inheritance, and that composition of such agents is the key to building practical, cost-effective AI systems. He defines an agent as deterministic software harnessing non-deterministic model outputs, then critiques the current approach of piling tools and skills into a single agent (inheritance), proposing instead a coordinator agent that delegates to many specialized sub-agents (composition), each with its own minimal context, system prompt, and tools. He claims these domain-specific agents achieve over 80% token efficiency, enable use of cheaper small models (e.g., DeepSeek V4 Flash is 137x cheaper than a frontier model), improve safety by limiting capabilities, and scale easily. Schroeder predicts rapid adoption through 2026-2027, with multi-agent orchestration becoming mainstream and tokens no longer getting cheaper—up 29% when adjusted for IQ. He describes an ideal agent architecture with hooks, rules, and recursive sub-agents, and invites listeners to try…

How I deleted 95% of my agent skills and got better results — Nick Nisi, WorkOS

May 30, 2026 · 17:43

Nick Nisi, DX engineer at WorkOS, argues that AI agents should be forced to prove their work with code rather than trusted with prompts. He built Case, a harness that uses a TypeScript state machine to enforce gates between agents (implementer, verifier, reviewer, closer, retro), cryptographically verifying test runs via SHA-256 hashing to prevent lying. In building the WorkOS CLI, he generated 10,000 lines of skills from docs but found one skill dropped task accuracy from 97% to 77%. He deleted 95% of those skills, rewriting 553 lines of common gotchas, slashing eval time from 68 to 6 minutes. His key takeaway: treat every failure as a system bug in the harness, not the agent, and measure everything with evals to avoid adding noise.

Cooking with Agents in VS Code — Liam Hampton, Microsoft

May 21, 2026 · 17:04

Liam Hampton from Microsoft demonstrates how to run three AI agents simultaneously in VS Code using GitHub Copilot: a local agent with Claude Opus for iterative unit test writing, a background agent using git work trees to build a front end from a GitHub issue with minimal oversight, and a cloud agent in GitHub Actions to add documentation and open-source files. He argues that VS Code serves as a single entry point for all three agent types—local for hands-on iteration, background for large tasks where partial involvement is acceptable, and cloud for tasks the developer doesn't need to touch. The talk walks through a live demo where the three agents work in parallel on one codebase, solving different problems (testing, UI, documentation) without interference, and explains the underlying infrastructure: cloud agents run securely in GitHub Actions with MCP servers and built-in safeguards, while local and background agents leverage VS Code's chat customizations and third-party extensions. Hampton also highlights the new modal for managing agents, custom instructions, skills, and MCP servers, positioning Copilot as a unified control plane for diverse AI workflows.

Anthropic Workshop: Build Agents That Run for Hours — Ash Prabaker & Andrew Wilson

May 18, 2026 · 1:15:40

Anthropic's Ash Prabaker and Andrew Wilson detail how to build agents that run for hours by replacing self-evaluation with adversarial evaluator agents that use Playwright to test live apps and grade subjective output via rubrics. They explain that context compaction doesn't cure coherence drift, so structured handoffs between fresh context windows are essential. The generator and evaluator negotiate testable sprint contracts before building, and the planner provides high-level specs without overspecifying technical details. They show that a solo Claude Code session built a retro game maker that looked complete but failed in play mode, while the adversarial harness produced a fully functional app with live physics and AI features. Key takeaways include reading traces as the primary debug loop, deleting harness components as models improve, and using file-system state for long-running agents.

Harnesses in AI: A Deep Dive — Tejas Kumar, IBM

May 17, 2026 · 20:27

Tejas Kumar, AI Developer Advocate at IBM, defines an AI harness as everything around the model that grounds it in reality, contrasting it with an agent loop. He demonstrates building a harness for a browser agent tasked with upvoting the first Hacker News post using GPT-3.5 Turbo, emphasizing that the prompt remains unchanged. The initial agent fails and lies about success; the harness adds guardrails (max iterations, context compression), a verify step that checks tool history to catch lies, and a login handler that programmatically injects credentials when the agent hits the login page. The final agent reliably logs in and upvotes. Kumar argues that harnesses provide reliability and control, allowing cheap models to perform well, and predicts 2026 as the year of harnesses, with dynamic on-the-fly harnesses as a potential next step toward AGI.

Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft

May 16, 2026 · 19:45

Marlene Mhangami of Microsoft and GitHub argues that AI boosts developer productivity only when paired with clean code practices, citing a Stanford study of 120,000 developers and GitHub Octoverse data showing 275 million weekly commits in 2026 with a growing share co-authored by AI. She advocates test-driven development (TDD) with Playwright for functional testing over unit tests, as AI-generated tests often affirm code behavior rather than user experience. In a live demo, she uses GitHub Copilot CLI and Playwright MCP server to automatically write failing Playwright tests for a toy store's search and filter features, then generate code to pass them. She emphasizes testing one feature per test, capturing screenshots for PRs, and committing code before fixes to preserve context. She closes with advice on using Playwright agents for complex state management and confirming cross-screen-size support.

Ralph Loops: Build Dumb AI Loops That Ship — Chris Parsons, Cherrypick

May 4, 2026 · 1:48:26

Chris Parsons argues that dumb loops—simple while loops where an AI agent repeatedly executes the same task, evaluates its output, and iterates—outperform complex multi-agent orchestration and planning graphs. In this hands-on workshop, he builds a Pomodoro timer using Claude Code, implementing tickets one by one in a loop that self-corrects with each pass. He demonstrates the 'loop' command in Claude Code for scheduled, continuous execution and shows how to use sub-agents for adversarial validation to avoid self-confirmation bias. The talk covers real-world applications such as a worker loop that processes project steps overnight and a morning loop that generates daily briefings, all designed to offload routine work while keeping human oversight for irreversible decisions. Parsons emphasizes that the bottleneck in AI-augmented teams is often the review process, not coding speed, urging teams to identify and fix their biggest constraint first.

Building your own software factory — Eric Zakariasson, Cursor

Apr 28, 2026 · 1:23:37

Eric Zakariasson, an engineer at Cursor, explains how to build a 'software factory' by scaling from one agent to many, shifting from worker to manager. He outlines six stages of autonomy, from spicy autocomplete to the dark factory, and emphasizes that most teams are stuck at levels two or three. Key primitives include modular codebases, guardrails like rules and hooks that emerge dynamically, enablers such as skills and MCPs, and verifiable systems like automated tests. Cloud agents with isolated VMs allow agents to run asynchronously, test their own work via computer control, and scale to thousands. Eric shares internal automations like daily reviews, PR comment analysis, and continual learning that extracts rules from chat transcripts. He concludes with strategic advice: front-load context, don't outsource critical decisions, and build tools and systems to capture flywheels, noting that human accountability remains essential.

Full Walkthrough: Workflow for AI Coding — Matt Pocock

Apr 24, 2026 · 1:36:30

Matt Pocock presents a hands-on workshop on building a full AI-assisted coding workflow, arguing that software engineering fundamentals—not hype—make agents effective. He introduces the 'smart zone' and 'dumb zone' of LLMs (performance drops after ~100k tokens) and the 'Memento problem' (agents forget between sessions). His process starts with a 'Grill Me' skill that relentlessly questions the user until shared understanding is reached, then produces a PRD without reading it, slices work into vertical 'tracer bullet' issues, and runs agents AFK using TDD. He advocates designing codebases with deep, testable modules and shows Sandcastle, a TypeScript library for parallel agent execution with separate implementer (Sonnet) and reviewer (Opus). The workshop transforms ambiguous briefs into shippable features while keeping humans in the loop for QA and taste.

AI Didn’t Kill the Web, It Moved in! — Olivier Leplus (AWS) & Yohan Lasorsa (Microsoft)

Apr 10, 2026 · 52:55

Yohan Lasorsa (Microsoft) and Olivier Leplus (AWS) argue that AI isn't replacing the web—it's becoming its native layer, embedded in every stage of development. They demo coding agents that use lightweight 'skills' to automate workflows like fetching GitHub issues and testing via Playwright, then show Chrome's MCP server letting agents debug performance across different network conditions. New on-device Web AI APIs (summarizer, proofreader, prompt API) run models locally in the browser, enabling features like auto-writing reviews from an uploaded product image without cloud calls. Finally, they introduce LLMs.txt for guiding AI agents to relevant documentation and Web MCP—a proposal to register native tools on websites (e.g., 'add to cart') so agentic browsers can execute actions without mimicking human clicks. The episode delivers concrete implementations, from coding to deploying AI-ready web apps.

Bending a Public MCP Server Without Breaking It — Nimrod Hauser, Baz

Apr 8, 2026 · 40:50

Nimrod Hauser, founding engineer at Baz, presents a hands-on guide to adapting third-party MCP servers for production, using Playwright's MCP server as an example. He demonstrates five best practices—curation, wrapping descriptions, deterministic guardrails, composing new tools, and treating tools as deterministic functions—to transform a brittle setup into resilient infrastructure. Hauser shows how curating tools from 21 to 16 reduces context load, while enhanced descriptions guide agents to use accessibility snapshots before clicking. Deterministic path validation prevents agents from saving screenshots outside a designated folder, and a new 'evidence tool' with tailored instructions improves spec review accuracy. The talk culminates with Baz's Spec Reviewer, an agentic code review tool that compares requirements against implementation, successfully passing a test after applying all optimizations.

Developer Experience in the Age of AI Coding Agents – Max Kanat-Alexander, Capital One

Dec 23, 2025 · 18:20

Max Kanat-Alexander, Capital One executive distinguished engineer, argues that developer experience teams should invest in a 'No Regrets' framework to prepare for an agentic future, regardless of unpredictable AI advancements. He identifies six key inputs that benefit both humans and AI agents: standardized development environments using industry-standard tools, native CLIs and APIs instead of browser automation, deterministic validation with clear error messages, refactoring legacy codebases for testability and reasoning, documenting external context and intentions, and improving code review velocity and quality through assignment and apprenticeship. Without these, bad codebases lead to rubber-stamped PRs and a vicious cycle of decreasing agent productivity; with them, a virtuous cycle accelerates software engineering velocity. Kanat-Alexander emphasizes that what's good for humans is good for AI, and these investments will help developers no matter what the future holds.

The 3 Pillars of Autonomy – Michele Catasta, Replit

Dec 22, 2025 · 24:42

Michele Catasta, VP of AI at Replit, argues that true autonomy for coding agents serving non-technical users rests on three pillars: verification, context management, and parallelism. Replit’s agent uses autonomous testing—writing Playwright code instead of browser-use tools—to catch broken features (over 30% initially) without human feedback, cutting cost and latency by an order of magnitude. Context management relies on sub-agent orchestration rather than massive context windows, boosting memories per compression from ~35 to 45–50. Parallelism, implemented via a core-loop orchestrator, dynamically decomposes tasks to avoid merge conflicts and keep users engaged instead of waiting hours. Catasta emphasizes that autonomy means scoped technical decisions, not just long runtimes, enabling knowledge workers to build software without needing a 'driving license.'

Proactive Agents – Kath Korevec, Google Labs

Dec 13, 2025 · 16:51

Kath Korevec, Director of Product at Google Labs, argues that AI coding agents must become proactive rather than reactive to truly reduce developer cognitive load. She introduces Jools, a proactive autonomous coding agent that observes workflows, personalizes responses, and intervenes at the right moment. Korevec details three levels of proactivity: level one auto-fixes issues during tasks; level two learns project context; level three connects agents across code, design, and data. She highlights features like memory, a critic agent for code quality, verification via Playwright, and a to-do bot. A demo shows Jools indexing a codebase and suggesting high-confidence tasks. Korevec ties this to her personal Halloween animatronic project, where she wished Jools handled debugging so she could focus on creative LED animations.

Real World Development with GitHub Copilot and VS Code — Harald Kirschner, Christopher Harrison

Aug 3, 2025 · 1:19:33

Harald Kirschner demonstrates three stages of Vibe Coding with GitHub Copilot and VS Code—YOLO, Structured, and Spectrum—arguing that developers can move from rapid prototyping to maintainable, scalable AI-assisted development. He shows how to use Agent mode, auto-approve settings, and new workspace scaffolding to build a hydration tracking app with React Vite and Material Design without looking at code. The talk covers custom instructions, MCP servers (Playwright, Jest Pad, Perplexity), and custom modes like TDD that enforce test-driven development by writing failing tests first. Kirschner explains how to constrain tools for deterministic behavior using tool sets and recommends committing often, pausing AI when needed, and iterating on instructions. He also discusses spec-driven development, critiquing specs with AI, and the importance of well-structured codebases for AI productivity.

Piloting agents in GitHub Copilot - Christopher Harrison, Microsoft

Jul 26, 2025 · 59:07

Christopher Harrison (Microsoft) demonstrates GitHub Copilot's agent capabilities for AI Engineers, arguing context is key to effective pair programming. He explains Copilot's modes—completion, chat, edit, local agent, and coding agent—and how Model Context Protocol (MCP) enables access to external tools like databases. Harrison shows how Copilot-instructions.md files provide project-specific guidance, and how coding agent runs within ephemeral GitHub Actions environments with strict security (no external access, only write to its own branch). He emphasizes that AI doesn't change DevOps fundamentals: tests still run locally and via PR workflows, and code review remains essential. The lab covers assigning issues to Copilot, configuring MCP servers, and using .instructions files for reusable patterns, with a repeat session at 3:30 PM.

Your Coding Agent Just Got Cloned And Your Brain Isn't Ready - Rustin Banks, Google Jules

Jul 25, 2025 · 13:40

Rustin Banks, an AI product manager at Google Labs, shares how Jules, Google's asynchronous coding agent powered by Gemini 2.5 Pro, enables developers to shift from sequential coding to orchestrating many parallel agents. He demonstrates real-world use on a conference schedule website, adding tests with Jest and Playwright simultaneously, implementing features like Google Calendar integration and AI summaries, and running accessibility audits. Banks argues that success requires clear verification criteria, creating 'disagreement' with agents to ensure correctness, and a robust merge/test framework. He notes that in its first two weeks, Jules generated 40,000 public commits, and emphasizes that agents-as-a-service in the cloud allow infinite scalability and development from any device.

Full Spec MCP: Hidden Capabilities of the MCP spec — Harald Kirschner, Microsoft/VSCode

Jul 18, 2025 · 14:53

Harald Kirschner from Microsoft/VSCode argues that MCP's full specification unlocks powerful stateful interactions beyond the common 'tools-only' implementations, transforming AI assistants into more contextual and efficient agents. He highlights underused primitives like resources for rich data context and sampling for server-requested LLM completions, demonstrated via a dungeon game where dynamic tool discovery adapts to game state. VS Code's upcoming full spec support includes dynamic tool discovery, user-defined tool sets, a debug mode for server development, and support for streamable HTTP to reduce stateful server churn. Upcoming features like elicitations will allow tools to request user input directly. Kirschner calls on developers to build progressive, full-spec servers and contribute feedback to the open ecosystem, emphasizing that client and SDK support will follow as usage grows.

The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp

Jun 30, 2025 · 35:06

Beyang Liu, CTO and co-founder of Sourcegraph, argues that coding agents are a real and high-ceiling skill, contrary to skeptics like Jonathan Blow and Eric S. Raymond. He presents design decisions for the agentic era: agents should directly edit files instead of asking permission, UIs should be minimal (like Amp's bare-bones VS Code extension and CLI), and fixed pricing should give way to usage-based models. In a live demo, Liu uses Amp to implement a custom icon for the linear connector in Amp's own codebase, demonstrating agentic search and sub-agents. He shares power-user patterns, including writing long prompts, constructing feedback loops with Playwright and Storybook, and running multiple agents in parallel—as exemplified by Jeff Huntley using Amp to build a compiler while sleeping. Liu emphasizes that agents should enable more thorough code reviews, not replace human understanding.

[Full Workshop] Vibe Coding at Scale: Customizing AI Assistants for Enterprise Environments

Jun 27, 2025 · 1:20:38

Harold from the VS Code team demonstrates "Vibe Coding at Scale" at the AI Engineer World's Fair, presenting three stages—YOLO, structured, and spectrum vibes—for customizing AI assistants in enterprise environments. He live-codes a hydration tracking app using GitHub Copilot's agent mode, auto-approve, and new workspace scaffolding, then shows how to enforce design principles via Copilot instructions and reusable prompts. Harold introduces custom modes (e.g., TDD mode), tool sets, and MCP servers like Playwright and Perplexity for research and browser testing. He emphasizes iterating on instructions, committing often, and using spec-driven development to balance speed with reliability, arguing these techniques enable true flow-state collaboration even on complex codebases.

Agentic Excellence: Mastering AI Agent Evals w/ Azure AI Evaluation SDK — Cedric Vidal, Microsoft

Jun 27, 2025 · 20:01

Cedric Vidal, Principal AI Advocate at Microsoft, demonstrates how to systematically evaluate AI agents using the Azure AI Evaluation SDK and AI Toolkit. He argues that evaluation must start early, covering four layers: model/safety system, system message, grounding, and user experience. He shows spot-checking models side-by-side in VS Code, building a web-scraping agent with Playwright MCP, and running batch evaluations with built-in quality and safety metrics like groundedness and fluency. Vidal also covers multi-turn and multimodal evaluation, using a violent image example where the SDK returned a score of 4 out of 5, allowing teams to set custom thresholds based on application context.

Building Agents with Amazon Nova Act and MCP - Du'An Lightfoot, Amazon (Full Workshop)

Jun 21, 2025 · 1:26:20

Du'An Lightfoot and Benjiro Byami present a workshop on building AI agents with Amazon Nova Act, Model Context Protocol (MCP), and Strands Agents, demonstrating autonomous web navigation and multi-step task execution. Lightfoot explains agentic AI as plan, action, and reasoning, with components like LLM, knowledge base, guardrails, and tools. Nova Act, a research preview, enables browser automation: one demo searches Amazon for a coffee maker and retrieves the product title using natural language commands instead of explicit HTML tags. Byami builds an MCP server for Nova Act, allowing clients like Claude Desktop to control the browser via spoken instructions, then integrates it with Strands Agents, an open-source framework that combines a prompt, LLM, and tools in minimal code. The workshop shows parallel execution (e.g., comparing three monitors) and multi-agent collaboration for tasks like generating a PowerPoint presentation for cloud migration. Limitations include inability to bypass CAPTCHAs and occasional loops in complex websites.

The Web Browser Is All You Need - Paul Klein IV, Browserbase

Jun 20, 2025 · 17:31

Paul Klein IV, founder of Browserbase, argues that the web browser is the default MCP server for the rest of the internet, serving as the essential bridge between AI agents and legacy websites that lack APIs. He distinguishes between web agents (one prompt to many actions, e.g., OpenAI's Operator) and browser tools (one action per prompt), noting that both rely on vision-driven or text-based approaches to parse pages. Klein explains that Browserbase offers a horizontal MCP server for browsing, enabling automation of any website, and stresses the need for custom evals and observability to track agent behavior. In a live demo, he shows an agent navigating sfpca.org to find a dog for adoption, handling unexpected modals. He also addresses captchas, advising good citizenship and hinting at future agent authentication solutions.

MCP Agent Fine tuning Workshop - Ronan McGovern

Jun 3, 2025 · 35:30

Ronan McGovern demonstrates how to fine-tune a Qwen3 model on high-quality reasoning traces from an MCP agent with browser tools. Using Playwright's 25 tools via Model Context Protocol, he generates multi-turn traces by running a 30B Qwen agent on RunPod, saving both tool calls and reasoning content. Those traces are then used to supervised fine-tune a 4B Qwen model with Unsloth, applying LoRA adapters to attention and MLP layers. The process includes converting MCP tool schemas to OpenAI format, extracting Hermes-style tool calls, and unrolling conversations to multiply training examples. Even with only nine curated traces, the fine-tuned model shows improved tool-calling behavior on multi-step tasks like navigating trellis.com to extract specific content.

The Current State of Browser Agents - Jerry Wu and Wyatt Marshall

Jun 3, 2025 · 21:13

Jerry Wu and Wyatt Marshall of Halluminate evaluate browser agents with WebBench, a benchmark of 5,000 tasks across 500+ websites. They find read tasks succeed at ~80%, but write tasks drop below 40% for autonomous agents, with failures often due to infrastructure like CAPTCHAs rather than agent intelligence. They advise AI engineers to choose read over write use cases, test multiple browser infrastructure providers, and combine agents with deterministic Playwright workflows for production. Notable emergent behaviors include an agent bypassing Cloudflare by searching Google and an agent posting a top-liked Medium comment.

This video was edited with AI agent. But how?

Feb 22, 2025 · 5:00

Muhteşem from Re-Skill presents the world's first open-source video editing agent, built in collaboration with Diffusion Studio, which uses LLM-generated code to edit videos via a browser-based UI. The agent starts a Playwright browser session and connects to operator.diffusion.studio, a web app built with Diffusion Studio Core that renders video directly in the browser using WebCodecs. Three tools drive the process: VideoEditingTool generates and runs JavaScript code based on user prompts, DocsSearchTool uses RAG to pull relevant information from llms.txt, and VisualFeedbackTool samples one frame per second and decides whether to proceed with rendering or refine further. File transfers between the Python agent and browser happen via Chrome DevTools Protocol, and for scalability, the agent can connect to a remote GPU-accelerated browser through WebSocket. The project emerged from Re-Skill's need for automatic video editing for personalized learning, where they found FFmpeg too limiting and turned to Diffusion Studio Core for its intuitive API and reliable in-browser rendering.