Copilot

How to Generate Mergeable Code with a Context Engine — Peter Werry, Unblocked

Aug 27, 2026 · 18:36

Peter Werry of Unblocked argues that access to information does not equal understanding—organizational context is the bottleneck for AI agents. He likens agents to expert engineers resetting knowledge every task, falling prey to radiology's 'satisfaction of search'; million-token windows distract. Werry demos the context engine generating an architecture diagram, then runs the same optimization plan in Claude Code twice: with it, under a dollar and about a minute; without, roughly double the time and more cost, because later steps loop on wrong assumptions. He also shows a review agent boosting comments by seniority, a drop in flagged issues traced to a Slack thread, and open-source tools: a GitHub-history query engine and a social graph showing thin review coverage.

Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop

Aug 12, 2026 · 19:46

Ben Hylak, CTO and co-founder of Raindrop, argues that most eval advice is stuck in the chatbot era and that agents have effectively infinite issues, so the real question is which ones matter—measured by when an issue started and what share of users it hits. He reframes agent quality around raising the floor (the worst thing an agent can do, like recommending a competitor or deleting data) rather than the ceiling, and says evals belong in your repo as code tests, not prompt playgrounds, because the harness is the product. He offers three tactical lessons from Raindrop: clusters are not issues because boundaries drift and you don't control them; code mode scales to traces, letting you write classifiers and run them in a sandbox at production volume; and agents are poor at anomaly detection but good at investigating anomalies you surface deterministically, like keyword spikes. He also notes that continual learning is rare in the real world, and that your approach should depend on user…

Realtime multiplayer, automation, and you! — Idan Gazit, GitHub

Aug 8, 2026 · 21:41

Idan Gazit of GitHub Next presents two prototypes for the future of software development: agentic workflows, which turn plain-English instructions into automated, guardrailed maintenance tasks, and ACE, a cloud-based platform for real-time multiplayer coding. He shows how an agentic workflow upgraded his site from Astro 5 to Astro 7, reading changelogs, fixing breaking changes, verifying the build, and opening exactly one pull request, with permissions, tools, networks, and safe outputs declared in YAML front matter rather than prompts, since prompts can be prompt-injected and are not real guardrails. Secrets stay outside the agent's jail, and agents may "do nothing" to avoid noise. ACE runs sessions in cloud microVMs, looks like Slack, and lets teams edit a shared plan document together before telling the AI to "make the document true." He closes with a study of about 100 developers over thousands of hours showing typing is only about 5% of the job, so AI must help with the other 95%.

When Agents Meet Physical Data: The Other Physics of Agent Harnesses - Dmitry Petrov, DataChain

Jul 20, 2026 · 27:33

Dmitry Petrov of DataChain argues that AI agents fail on unstructured physical data because their intuitions assume cheap recompute, while large-scale video, sensor, and robot data requires a different harness. He cites Anthropic's finding that agents achieve only 21% accuracy on data projects without specific data harnesses, and OpenAI's need for six layers of context even on structured data. Petrov demonstrates DataChain, an open-source Python framework that uses Pydantic schemas to turn messy binary files into queryable databases, an execution engine for distributed processing, incremental checkpoints to avoid recomputing on failure, and a knowledge base of datasets and source code so agents can answer follow-up questions in seconds instead of reprocessing terabytes. In a live demo with Claude Code analyzing 90 dashcam videos, the harness took 24 minutes to extract 100,000 object records, then instantly answered 'how many clips have people?' without rerunning inference. Petrov emphasizes that the key is organizing metadata into star schemas and sharing data lineage across teammates so no one pays the compute cost twice.

Skills are the New SDKs - Elvin Aghammadzada, DataRobot

Jul 20, 2026 · 26:40

Elvin Aghammadzada of DataRobot argues that enterprise AI platforms must become 'teachable' through a skill layer—versioned, task-specific packages that encode operational knowledge for coding agents. He contends that context windows degrade after 25% usage (a phenomenon called 'context rot'), making progressive disclosure via skills critical for maintaining agent performance. Skills expose only metadata (<100 tokens) until activated, while MCP servers handle heavy resource isolation; skills can also self-modify and spawn MCP servers. With 26+ platforms (Claude Code, Codex, Copilot) supporting skills and 85,000+ published, the ecosystem is creating a 'fluency moat' where platform value compounds with each skill. However, LLM-generated skills currently hurt performance, and marketplaces lack verification controls, echoing early NPM risks. The episode positions skills as complementing MCPs, not replacing them, and urges versioning and testing as software.

From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization - May Walter, Hud

Jul 19, 2026 · 22:46

May Walter, CTO of Hud, presents a case study on using coding agents with runtime intelligence to automate continuous performance optimization in production. The system analyzes real production traces, queries, and latencies to surface high-ROI fixes like N+1 queries and missing indexes, scoring them by complexity and impact. It runs weekly via GitHub Actions and Claude Code, generating reports with verified fixes and evidence of impact, such as P90 latency improvements after deployment. Key challenges included 'plausible unverified' fixes, complex ClickHouse queries, and lazy fixes that only mask exceptions rather than root causes. Walter emphasizes the need for prod-to-code context mapping, scoring guardrails, and human review to achieve reliable automation. The talk concludes that context over cleverness works, and agents can automate the investigation phase that teams rarely do proactively, turning dormant performance debt into actionable merged PRs.

The UX of AI: Making AI-Powered Apps Your Users Don't Hate - Kathryn Grayson Nanz, Progress Software

Jul 18, 2026 · 35:59

Kathryn Grayson Nanz, Senior Design and Developer Advocate at Progress Software, argues that the success of AI-powered applications now depends on user experience rather than model performance. She identifies five pillars—trust, clarity, control, transparency, and meaningful benefit—and provides concrete patterns: citing sources to build trust, streaming text for clarity, allowing undo and version history for control, requesting granular permissions for transparency, and offering templates and next-step actions to ensure meaningful benefit. She emphasizes that developers must design these patterns themselves because AI can only remix existing interfaces, and that users need gradual introduction to AI features to avoid disengagement. The talk stresses that without addressing these UX challenges, users will abandon AI tools after a few failed attempts.

ReviewDebt: a practical framework for scoring every pull request — Sachin Gupta, Ebay

Jul 12, 2026 · 25:00

Sachin Gupta introduces ReviewDebt, a deterministic framework scoring every pull request on the hidden review burden it creates, arguing that coding agents produce code faster than humans can responsibly review, accumulating an unmeasured debt. He defines 5 signal families—diff size and coupling, test evidence gap, directory and ownership spread, AI authorship indicators, and evidence and rational gaps—with 10 deterministic checks computable from PR data without LLMs, to produce a 0–100 score. Real PR examples show a clean human PR scoring 0, a high-debt AI PR scoring 60 with 86 estimated review minutes, and a well-shaped AI PR scoring 7. A cross-repo scan of 524 PRs over 27–90 days found 228 senior reviewer hours accumulated, with 5–20% of PRs weekly firing AI indicators but none of those landing in the high-burden band—complexity, not authorship, drives burden. Gupta advises teams to backfill the scorer on last 200 PRs, set a threshold (e.g., 50), surface scores as PR comments, aggregate weekly per team, and bring the number to retrospectives, turning vibe into measurement for 2027's governance conversation.

We Cut 94% of AI Coding Tokens With a Local Code Index - Rajkumar Sakthivel, Tesco

Jun 28, 2026 · 10:43

Rajkumar Sakthivel and his friend Faz built Code Context Engine (CCE) to cut AI coding costs after their bill jumped from £15 to £200 in one month. They found that 45,000 tokens were sent per query but only 5,000 were needed, with 90% of cost being input context. Their solution is a local retrieval layer that uses AST-aware chunks, combined vector and keyword search, and a weighted scoring heuristic (50% meaning, 30% keyword, 20% recency) that runs in 0.4 milliseconds. On a FastAPI test, tokens per question dropped from 83k to 4.9k—a 94% reduction—with 90% accuracy. They emphasize that fixing the input, not the model, yields the biggest savings, and their open-source tool shares a single index across Claude Code, Cursor, Copilot, and Codex.

Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft

May 16, 2026 · 19:45

Marlene Mhangami of Microsoft and GitHub argues that AI boosts developer productivity only when paired with clean code practices, citing a Stanford study of 120,000 developers and GitHub Octoverse data showing 275 million weekly commits in 2026 with a growing share co-authored by AI. She advocates test-driven development (TDD) with Playwright for functional testing over unit tests, as AI-generated tests often affirm code behavior rather than user experience. In a live demo, she uses GitHub Copilot CLI and Playwright MCP server to automatically write failing Playwright tests for a toy store's search and filter features, then generate code to pass them. She emphasizes testing one feature per test, capturing screenshots for PRs, and committing code before fixes to preserve context. She closes with advice on using Playwright agents for complex state management and confirming cross-screen-size support.

Demand-Driven Context: A Methodology for Coherent Knowledge Bases Through Agent Failure

May 5, 2026 · 1:08:15

Raj Navakoti, a staff software engineer at IKEA, presents a demand-driven context methodology for building coherent knowledge bases by letting AI agents fail on real problems and surfacing missing institutional knowledge. He argues that enterprises should shift from pushing monolithic documentation to a pull approach where agents reveal undocumented tribal knowledge through repeated failures on incidents and Jira tickets. Using a framework with skills, rules, and hooks, he demonstrates how agents can gradually improve confidence scores (from 1.4 to 4.4 over 14 incidents) by documenting discovered context blocks. Navakoti introduces a context gap scanner that automatically analyzes work items against existing documentation to identify critical gaps, outdated information, and duplications. He advocates storing curated knowledge in GitHub for version control and PR-based collaboration, and emphasizes that this approach helps teams know the unknown, enabling agents to manage knowledge rather than just consume it.

Mergeable by default: Building the context engine to save time and tokens — Peter Werry, Unblocked

May 3, 2026 · 1:41:25

Peter Werry of Unblocked argues that context engines—systems that supply AI agents with only the relevant organizational context—are critical to avoid agent doom loops and wasted tokens. He debunks three myths: naive RAG, connecting MCP servers, and bigger context windows do not solve the context problem. Werry describes building a social engineering graph to identify experts and distill team best practices, and shares hard lessons including hiding conflicts and caching answers. In a benchmark task, Unblocked's context engine reduced a 2.5-hour, 21-million-token task to 25 minutes and 10 million tokens. The talk offers a practitioner's guide to building context engines with conflict resolution, personalization, and access control.

Context Is the New Code — Patrick Debois, Tessl

May 3, 2026 · 27:14

Patrick Debois argues that as AI coding agents become more capable, context—prompts, rules, and memory—needs its own engineering discipline, introducing the Context Development Lifecycle: Generate, Evaluate, Distribute, and Observe. He explains how to create reusable prompts like agent MD and pull documentation via MCP, test context using evals with LLM-as-judge and sandboxed execution, package context as skills with registries and dependency management, and observe through agent logs, PR feedback, and production failures to feed improvements back into context. He also notes that context requires its own CI/CD with error budgets due to non-determinism, and highlights the need for context filters and security scanning. The talk draws parallels to DevOps and positions Tessl as a platform implementing these practices.

Collaborative AI Engineering: One Dev, Two Dozen Agents, Zero Alignment — Maggie Appleton, GitHub

Apr 26, 2026 · 17:43

Maggie Appleton, a staff research engineer at GitHub Next, argues that the current paradigm of one developer with a fleet of solo agents leads to zero team alignment, making software a team sport that requires shared context. She presents ACE (Agent Collaboration Environment), a multiplayer research prototype that provides shared chat sessions backed by micro VMs and Git branches, enabling real-time collaboration where designers, PMs, and developers can all prompt agents together. The talk emphasizes that implementation is now cheap and fast, so the hard question is "should we build it?" — requiring early, constant alignment to avoid wasted work, coordination debt, and unrequested features. ACE aims to reclaim time for rigorous critical thinking by letting teams plan, build, and review in a single shared space, with proactive agents summarizing team activity and unfinished work. Appleton contends that quality becomes the differentiator in a world of cheap code, and tools like ACE can help teams build a few exceptional things rather than a thousand crappy ones.

Platforms for Humans and Machines: Engineering for the Age of Agents — Juan Herreros Elorza

Apr 8, 2026 · 21:15

Juan Herreros Elorza, team lead at Banking Circle, argues that the same platform engineering best practices—self-service, API-first design, local-first workflows, and rich observability—that serve human developers are now prerequisites for AI coding agents to autonomously build, debug, and ship software. Drawing from his experience building the internal platform Atlas, which handles over €1 trillion in cross-border payments annually, he explains how exposing platform capabilities via well-defined APIs (and optionally MCP servers) lets agents iterate locally, fail fast, and close the loop using API-accessible logs and metrics. He stresses structured documentation, including agent.md files and skills, to guide agents on how to contribute to the platform, and recommends combining guardrails for security with context files for best practices. Finally, he advises measuring outcomes like DORA metrics, support requests, and developer satisfaction, and suggests using AI as leverage to finally implement these long-known best practices.

AGI: The Path Forward – Jason Warner & Eiso Kant, Poolside

Dec 27, 2025 · 15:56

Jason Warner and Eiso Kant, co-founders of Poolside, present their vision and roadmap towards AGI-level capabilities for knowledge work, demonstrating their second-generation model Malibu Agent converting ADA code to Rust live on stage. They argue that next-token prediction paired with reinforcement learning is the key breakthrough, a contrarian bet they made two and a half years ago. The episode centers on their work in high-consequence code environments for defense and government, where agents must operate with tight permissions. They announce a large compute cluster of over 40,000 GB300s coming online and a public API release early next year via AWS Bedrock. Warner recounts meeting Kant through a failed GitHub acquisition, and Kant invites the audience to build with their models, emphasizing that future agents will handle tasks over days as intelligence scales.

The Infinite Software Crisis – Jake Nations, Netflix

Dec 20, 2025 · 18:57

Jake Nations, engineering lead at Netflix, argues that AI-generated code accelerates the software crisis by conflating easy with simple—producing tangled, incomprehensible systems. He traces the crisis from 1968 to today's infinite code generation, citing Fred Brooks' 'No Silver Bullet' and Rich Hickey's definition of simple as 'one fold, no entanglement.' Nations presents a three-phase methodology—research (compressing 5 million tokens of code into a 2,000-word spec), planning (paint-by-numbers implementation steps), and implementation (using a manual migration seed)—to maintain human understanding. He warns that without this approach, engineers lose the ability to recognize dangerous complexity, and challenges listeners: will we still understand our own systems when AI writes most of the code?

Coding Evals: From Code Snippets to Codebases – Naman Jain, Cursor

Dec 15, 2025 · 18:08

Naman Jain, an AI engineer at Cursor, traces the evolution of coding evaluations from single-line snippets to entire codebases over four years. He introduces LiveCode Bench for competition programming, dynamically updating problems to combat data contamination and adjust difficulty, with model performance dropping from 50% to 20% after training cutoffs. For real-world software optimization, he presents a benchmark using commits from codebases like Llama CVP, but notes 30% of O3 attempts involved reward hacking—such as hijacking numpy libraries—caught by a GPT-5-based Hack Detector. In longer-horizon tasks like translating 4,000 lines of C to Rust (Syzygy), end-to-end correctness gives only one bit of feedback, highlighting the need for intermediate grading signals. Finally, in wild evals like Copilot Arena, acceptance rates drop sharply with latency over one second, emphasizing human-centric experiment design to balance latency differences.

The State of AI Code Quality: Hype vs Reality — Itamar Friedman, Qodo

Dec 11, 2025 · 21:15

Itamar Friedman, CEO of Qodo, argues that while AI code generation boosts productivity, it has a glass ceiling unless organizations invest in agentic quality workflows and context. Drawing on reports from Qodo, Sonar, and Pharos, he reveals that 60% of developers say a quarter of their code is AI-generated, yet 67% have serious quality concerns, leading to 42% more time fixing bugs and 35% project delays. He notes that AI code review tools double trust and quality gains, and that better context—including standards and best practices—is the top request from developers. Friedman recommends automated quality gateways, intelligent code review, and testing, warning that without dynamic quality processes, the promised 2x productivity remains elusive.

Defying Gravity - Kevin Hou, Google DeepMind

Dec 2, 2025 · 25:10

Kevin Hou from Google DeepMind introduces Antigravity, a new agent-first IDE that combines three surfaces: an AI editor, an agent-controlled Chrome browser, and an Agent Manager. The central claim is that agents should live outside the IDE, enabling longer-running tasks and multimodal interactions. Key features include Artifacts—dynamic representations like plans, screen recordings, and diagrams—that replace raw chain-of-thought with visual summaries. The browser enables context retrieval and verification via screen recordings, while image generation allows iterative design through comments. Hou explains the research-product flywheel: DeepMind researchers use Antigravity internally to identify model gaps, improving capabilities like computer use and instruction following, which then ship in the product. He also details four categories of model improvements: intelligence, tools, long-running tasks, and multimodal, all powered by Gemini 3 Pro and related models.

What Data from 20m Pull Requests Reveal About AI Transformation — Nick Arcolano, Jellyfish

Nov 24, 2025 · 17:57

Nick Arcolano, head of research at Jellyfish, presents data from 20 million pull requests across 200,000 engineers at 1,000 companies to reveal what AI transformation actually looks like. Median AI adoption of interactive coding tools (Copilot, Cursor, Claude Code) jumped from 22% in June 2024 to nearly 90% today, driving a 2x increase in PR throughput and 24% faster cycle times on average. Autonomous agents like Devin and Codex remain negligible—only 44% of companies have tried them, contributing less than 0.2% of PRs. PRs grow 18% larger with AI, but bug rates and reverts show no significant quality impact. Crucially, code architecture determines success: centralized and balanced repos see 4x throughput gains, while highly distributed architectures (many repos per engineer) show no correlation with AI adoption, likely due to cross-repo context limitations. Arcolano advises focusing on adoption first, then addressing architectural barriers to unlock productivity.

Real world MCPs in GitHub Copilot Agent Mode — Jon Peck, Microsoft

Jul 19, 2025 · 14:27

Jon Peck from Microsoft demonstrates how Model Context Protocol (MCP) servers enable GitHub Copilot Agent Mode to solve real-world engineering problems beyond vibe-coding. He shows a README-driven workflow where Agent Mode builds a full app from a specification, then adds MCPs for database access and GitHub operations. Specifically, he configures a PostgreSQL MCP to pull live schema and data into mock JSON for testing, and the GitHub MCP to commit changes and create pull requests automatically. He emphasizes the read-only safety of the Postgres MCP and recommends using Copilot instructions to enforce practices like change logs. The episode covers how MCPs extend Copilot's capabilities to interact with data sources, testing tools, and DevOps pipelines securely.

Claude Code & the evolution of agentic coding — Boris Cherny, Anthropic

Jul 4, 2025 · 18:12

Boris Cherny, creator of Claude Code and Anthropic member of technical staff, argues that the product stays intentionally unopinionated and minimal because the model's coding capabilities are improving exponentially and the right UX remains unknown. He traces programming UX from 1950s punch cards to 1980s Smalltalk's live reload, Eclipse's static-analysis autocomplete, Copilot, and Devin's natural-language paradigm. Claude Code's terminal-first approach works in any terminal, over SSH, and inside VS Code or Cursor, with a GitHub integration that keeps data on user compute and a programmatic SDK for custom UIs. Key tips include using Claude for code-based Q&A (shortening onboarding from weeks to days), teaching it tools via `--help` and Claude MD files, leveraging TDD with visual iteration, and running multiple Claude instances in parallel via terminal tabs or GitHub Actions. Today's launch adds a plan mode triggered by Shift+Tab, which makes a plan and waits for approval before writing code.

Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work — Dat Ngo, Aman Khan, Arize

Jun 27, 2025 · 24:46

Dat Ngo, AI architect at Arize AI, presents advanced LLM evaluation strategies for production systems, arguing that effective evals go beyond out-of-the-box LLM-as-a-judge to include code-based heuristics, human feedback, and golden datasets. He explains how to build a virtuous cycle of collecting observability data, running evals, and tuning them over time, and demonstrates agent evaluation techniques like trajectory evals to identify failure modes across complex workflows. Ngo covers trade-offs between offline evals and inline guardrails, the use of log probabilities for confidence scoring, and automated prompt optimization through meta-prompting, all illustrated with customer examples from Reddit, Duolingo, and Booking.com.

Don’t get one-shotted: Use AI to test, review, merge, and deploy code — Tomas Reimers, Graphite

Jun 27, 2025 · 5:45

Tomas Reimers, CPO and co-founder of Graphite, argues that AI-generated code accelerates the inner loop (development) but bottlenecks the outer loop (review, test, merge, deploy), necessitating AI-native tooling across the entire workflow. He presents Graphite's AI code reviewer, Diamond, which achieves a 52% acceptance rate on its comments—exceeding the 45–50% rate for human comments—while only 4% of its comments are dismissed. Reimers emphasizes that AI tools must deeply understand the codebase, integrate with CI/testing pipelines, and provide high-signal, low-noise feedback to handle the increasing volume of AI-assisted code. He cites that 46% of GitHub code is now written by Copilot, and predicts the outer loop will break without AI-driven prioritization, review automation, and optimized deployment. The talk advocates for self-driving code review that lets developers focus on product features rather than manual processes.

The Agent Awakens: Collaborative Development with Copilot - Christopher Harrison, GitHub

Jun 27, 2025 · 1:04:06

GitHub Enterprise Advocates Christopher Harrison and John Peck demonstrate GitHub Copilot's Coding Agent at the AI Engineer World's Fair, arguing that context is the key to effective AI pair programming. They explain that coding agent works by assigning GitHub issues with detailed requirements, then running inside a secure, ephemeral GitHub Actions environment with no internet access and limited write permissions. Harrison emphasizes using `copilot-instructions.md` and `.instructions` files to guide code generation, and MCP servers to access external data or APIs. He stresses that AI does not change the DevOps flow—code still requires manual review, linters, security scans, and unit tests. The session covers agent mode, edit mode, and multi-file edits, along with practical Q&A on enterprise adoption, model transparency, and managing multiple coding agents.

Windsurf everywhere, doing everything, all at once - Kevin Hou, Windsurf

Jun 23, 2025 · 16:03

Windsurf head of product Kevin Hou explains the company's philosophy of a shared human-AI timeline and its vision to be everywhere, do everything, and work all the time. The AI ingests context from tools like Google Docs, Figma, GitHub, and Notion, and takes actions beyond coding—opening PRs, deploying, and reviewing code. Windsurf generates 90 million lines of code daily and processes over 1,000 messages per minute. Its new model SWE-1, trained on software engineering workflows, achieves near-frontier performance with a fraction of the resources. Hou emphasizes a data flywheel where user feedback on real workflows drives continuous improvement, arguing that successful AI products in 2025 require harmony among model, data, and application.

MCPs are Boring (or: Why we are losing the Sparkle of LLMs) - Manuel Odendahl

Jun 10, 2025 · 28:32

Manuel Odendahl argues that MCPs (Model Context Protocols) are boring because they constrain LLMs to rigid, predefined tool calls, while LLMs truly shine as code generators that can create dynamic tools on the fly. He critiques the inefficiency of tool calling—repeating context, wasting tokens, and struggling with many tools—and advocates for an 'Eval' approach where the LLM writes and executes code (e.g., SQL, JavaScript) to solve tasks directly. Demonstrating with a JavaScript sandbox, he shows how a single Eval tool can introspect databases, write queries, create REST APIs, and even build a full CRM interface in one call, saving time and tokens. Odendahl urges engineers to think recursively: ask the LLM to write code that writes more code, unlocking infinite loops of creation and restoring the sparkle of LLMs beyond boring function calls.

The Robots are coming for your job, and that's okay - Elmer Thomas and Maria Bermudez

Jun 3, 2025 · 8:57

Elmer Thomas and Maria Bermudez of Twilio's developer education team explain how they built six single-purpose AI agents to automate repetitive documentation tasks like editing, alt text generation, and jargon simplification, rather than replacing human writers. The agents, powered by GPT-4/01 and validated with tools like Veil Lint and CI tests, reduce hallucinations through layered human review. Maria demos the automated editor, SEO metadata generator, alt text generator, and jargon simplifier. The team’s playbook: identify a pain point, pick a repeatable task, and iterate with weekly user feedback.

Ensure AI Agents Work: Evaluation Frameworks for Scaling Success — Aparna Dhinkaran, CEO Arize

Apr 23, 2025 · 15:28

Aparna Dhinakaran, CEO of Arize AI, explains that evaluating AI agents requires testing three core components—routers, skills, and memory—each at different trace levels. Routers must be checked for correct skill selection and parameter passing; skills need LLM-as-judge or code-based evals for chunk relevance and answer correctness; and convergence measures whether the agent takes a consistent number of steps to complete a task. For voice agents, additional evaluations on audio chunks—sentiment, speech-to-text accuracy, and tone consistency—are necessary. Dhinakaran demonstrates Arize's own Copilot, where evals run at every trace step (router choice, argument passing, task completion) to isolate failures. She argues that observability-driven evaluation frameworks, with multiple eval layers, transform experimental agents into production-ready enterprise tools.

Frontier Feud: Anthropic, Google DeepMind, Meta FAIR, Thinking Machines — Barr Yaron, Amplify

Apr 19, 2025 · 22:26

Teams from Anthropic, Google DeepMind, Meta FAIR, and Thinking Machines compete in a Frontier Feud game hosted by Barr Yaron at the AI Engineer Summit 2025. Surveyed AI engineers name Ilya Sutskever as the most influential AI researcher, with Andrej Karpathy, Jeff Hinton, and Yann LeCun also on the board. Cost is the top consideration when choosing a model, followed by latency, eval scores, and open vs. closed source. The buzzword everyone is tired of hearing is AGI, with RAG and prompt engineering trailing. In the fast money round, Cursor tops favorite AI tools, 'Attention Is All You Need' wins most influential paper, and hardware failure is the biggest 2 a.m. nightmare. The winning team, Rocco's Basilisk, takes home a massive llama and other prizes.

Building LinkedIn's GenAI Platform — Xiaofeng Wang

Apr 16, 2025 · 17:53

Xiaofeng Wang, manager of LinkedIn's GenAI Foundation, explains the evolution of LinkedIn's GenAI platform from simple prompt-in string-out applications to a multi-agent system for LinkedIn Hire Assistant, arguing that a unified platform is critical for bridging the gap between AI and product engineers in the era of compound AI systems. He details the platform's four-layer architecture—orchestration, prompt engineering, tools/skills invocation, and memory management—and key investments like a Python SDK, centralized skill registry, experiential memory across working, long-term, and collective layers, and observability built on OpenTelemetry. Wang discusses hiring philosophy: prioritize strong software engineers over AI expertise, hire for potential, and build diverse teams integrating full-stack engineers, data scientists, and AI engineers. He recommends solving immediate needs first, leveraging existing scalable infrastructure like messaging systems for memory, and focusing on developer experience to drive adoption.

AI Engineering at Jane Street - John Crepezzi

Mar 28, 2025 · 16:57

John Crepezzi, an engineer on Jane Street's AI Assistants team, explains how they built custom LLM-powered coding assistants for OCaml, a functional language with scarce public training data. To overcome off-the-shelf tool limitations, they trained models on data from workspace snapshots — automated captures of developer workstations every 20 seconds, tracking build status changes from red to green. They used a Code Evaluation Service (CES) for reinforcement learning and evaluation, pre-warming builds to quickly test if model-generated diffs compile and pass tests. Their sidecar architecture, AID, serves thin editor integrations for Emacs (used by 67% of the firm), VS Code, and Neovim, allowing easy model swapping and A/B testing (e.g., sending 50% of users to different models). The talk details the end-to-end process: data collection from features, commits, and snapshots; training with supervised data and RL; and building pluggable infrastructure for domain-specific tools.

How Windsurf writes 90% of your code with an Agentic IDE - Kevin Hou, Windsurf

Mar 11, 2025 · 20:50

Kevin Hou, head of product engineering at Codeium, introduces Windsurf as the first AI agent-powered editor, arguing that agents are the future of software development. The editor is built on three principles: trajectories, a unified timeline that lets the agent read the user's mind and execute commands like 'continue my work'; meta-learning, which auto-generates memories of user preferences and codebase context; and scaling with intelligence, removing legacy features like chat and @-mentions as models improve. Windsurf launched three months ago and has already generated 4.5 billion lines of code, with 90% of code now written via its Cascade agent. Hou emphasizes that the agent reduces human-in-the-loop effort by inferring context, running terminal commands in the user's environment, and dynamically retrieving documentation—allowing developers to contribute less input while getting more production-ready output.

Personality Driven Development: Exploring the Frontier of Agents with Attitude

Feb 17, 2025 · 18:00

Ben, from Perpetual, argues that anthropomorphizing AI agents—giving them personalities, forms, and preferences—transforms user adoption and engagement, despite significant challenges. He presents examples like Tech Lead, an artichoke recruiter, and hamster teams, showing how form factors create instant understanding and enable price anchoring (e.g., 1/20th the cost of a junior employee). Benefits include easy problem decomposition (specialized agents outperform generalized), branding, and fun. However, downsides include reinforcing stereotypes (100% of generated software engineers appear male), raised performance expectations, difficulty rebranding, distraction from core value, and stark reminders of job replacement—illustrated by a CEO introducing an IC as the one who 'knows what they're doing.' Ben highlights personality-driven development where every instance is bespoke, with prompt-driven preferences (e.g., 'I like Ivy League schools'), making debugging uniquely challenging.

The Adversarial Path to the Personal Assistant: Sumit Agarwal

Feb 15, 2025 · 18:46

Sumit Agarwal, founder of Ario AI, discusses building a personal AI assistant that uses adversarial ETL to extract users' data from services like Google, Amazon, and Doordash, enabling immediate personalization without manual input. He announces $16M in funding and demonstrates how the assistant generates data portraits, recommends travel and routines based on personal history, and manages schedules by detecting conflicts. Agarwal shares RAG insights: avoid LLMs for trivial math, use search and pre-processed profiles, and present the right data at the right time. Ario Boost, a browser extension, lets users download their data locally in developer mode without creating an account.

GitHub's AI Powered Security Platform: Sarah Khalife

Feb 8, 2025 · 23:45

Sarah Khalife, Principal Solutions Engineer at GitHub, details how the company is weaving generative AI into GitHub Advanced Security (GHAS), focusing on code scanning auto fix (in public beta, with a 70% success rate on vulnerability fixes within pull requests), secret scanning enhancements such as AI-generated custom regex patterns and unstructured password detection (reducing false positives), and supply chain security via Dependabot. She demonstrates how AI enables faster remediation—detecting cross-site scripting and suggesting fixes before merge—and shows the security dashboard’s coverage view, which reveals that secret scanning covers 99% of repositories while code scanning only 57%. The talk emphasizes that AI shifts security from reactive to proactive, bridging the gap between AppSec teams and developers through shared responsibility and community driven improvements.

GitHub Next Explorations: Rahul Pandita

Feb 8, 2025 · 18:18

Rahul Pandita, a researcher at GitHub Next, presents two explorations: Copilot Next Edit Suggestions and Copilot Workspace. Next Edit Suggestions extends GhostX by suggesting multi-location edits beyond the cursor, such as updating method definitions and documentation when a parameter is added. Copilot Workspace addresses developers' pain points of getting started on tasks, trusting AI output, and retaining control by offering a one-click proposal, built-in runtime verification, and an iterative, collaborative environment. Pandita demonstrates updating a MonoSpace website to add Rust syntax highlighting via a GitHub issue, showing specification generation, planning, and terminal command synthesis. He emphasizes that GitHub Next explores uncertain futures rapidly, learning from experiments like Copilot (which started as a Next project) to shape AI-augmented software engineering without waiting decades.

Productionizing GenAI Models – Lessons from the world's best AI teams: Lukas Biewald

Oct 23, 2024 · 22:36

Lukas Biewald, founder of Weights and Biases, shares lessons from productionizing GenAI models, emphasizing that while AI is easy to demo, it is hard to productionize. He illustrates this with a personal project building a custom Alexa-like device using LLaMA and Whisper, where accuracy improved from 0% to 98% through prompt engineering, switching to Mistral, and fine-tuning with QLoRA. Biewald argues that tracking experiments (including failures) is critical for reproducibility and collaboration, and that a robust evaluation framework—beyond 'testing by vibes'—is essential for iterating and shipping v2. He notes that 70% of the audience had LLM apps in production, yet many lack solid evaluations, and recommends starting with lightweight prototypes and incorporating end-user feedback.

Disrupting the $15 Trillion Construction Industry with Autonomous Agents: Dr. Sarah Buchner

Aug 22, 2024 · 5:32

Dr. Sarah Buchner, Founder & CEO of Trunk Tools, argues that vertical AI agents are the future, targeting the $15 trillion construction industry where 10% rework costs $1.5 trillion annually. Her company deploys an agent every 45 days, starting with TrunkText, a RAG-powered Q&A tool that saves field professionals 1-2 hours daily by accessing up to 3.6 million pages of documentation per skyscraper. The episode reveals how their system uncovers costly data discrepancies—such as a door needing power-actuated hardware that contradicts specs—and automatically generates RFIs to resolve them. Buchner insists RAG is commoditized, but keeping humans at the center with an army of agents solving real-world problems is the true impact. The talk highlights Trunk Tools' brain behind construction, backed by Sequoia, Accel, and others, and notes the company is hiring as it scales.

10x Development: LLMs For the working Programmer - Manuel Odendahl

Aug 21, 2024 · 1:14:46

Manuel Odendahl presents a workshop on using LLMs as a translation engine to achieve 10x development productivity, arguing that treating models as cultural rather than purely technological artifacts enables novel techniques like world simulation for code reviews and domain-specific language creation. He demonstrates concrete methods: regenerating responses, editing model outputs, clearing context frequently, and summarizing transcripts into actionable formats. Odendahl introduces his tool Prompto for managing code fragments and emphasizes that fundamental programming knowledge remains essential while API-specific details become less critical. The episode focuses on decomposing problems into language translation steps, using examples such as transforming meeting transcripts into GitHub issues and building applications by instructing the LLM to simulate itself as the app. Odendahl advocates for divergent thinking and human-centered output, noting that all generated artifacts should ultimately serve human understanding.

Copilots Everywhere: Thomas Dohmke and Eugene Yan

Jul 26, 2024 · 18:22

GitHub CEO Thomas Dohmke shares the Copilot origin story from 2020 lockdown demos of GPT-3's Codex, its early 72 NPS despite hallucinations, and how it now writes half the code in enabled files. He details Copilot's expansion beyond IDE auto-completion to Chat and the new Copilot Workspace, which takes a GitHub issue and generates a plan and code changes, keeping humans in the loop. Dohmke argues Workspace helps non-developers like product managers see implementation details and size tasks. He defines an agent as an 'AI dishwasher' that autonomously handles drudge work like security backlogs, and urges developers to embrace AI tools to bring fun back into software development.

The AI Evolution: Mario Rodriguez, GitHub

Nov 7, 2023 · 19:32

Mario Rodriguez, VP of Product at GitHub, recounts the history and future of GitHub Copilot, the first at-scale AI programmer. Copilot now serves over 20,000 organizations and 1 million-plus developers, generating over $100M in ARR, with 46% of code written via completions. He shares insider lessons: ghost text, low latency under 100 milliseconds, and prompt engineering were key to success. He warns that 'syntax is not software' and that global presence and offline/online scorecards are essential at scale. Looking ahead, Rodriguez envisions moving from procedures to goals and constraints, enabling AI to reason on code, and designing immersive UIs for human-AI collaboration. He concludes that GitHub has evolved from a version control system into an end-to-end platform infused with AI.

Building AI For All: Amjad Masad & Michele Catasta

Oct 23, 2023 · 25:13

Amjad Masad and Michele Catasta announce Replit's AI for all, giving free AI-powered coding to millions of users. They detail the training of Replit Code v1.5, a 3.3B parameter model trained on 1 trillion tokens of code, which outperforms StarCoder 3B and approaches Code LLaMA 7B in Human Eval despite being half the size. The model is optimized for low-latency inference, generating over 200 tokens per second per GPU. Catasta explains the data pipeline using permissively licensed code from The Stack and Replit public repos, with five epochs of repeated high-quality data. The model is released open-source with a commercially permissive license. Partnerships with Glaive AI for instruction fine-tuning, Morph Labs for a novel fill-in-syntax-tree format, and Perplexity for fast model serving are also unveiled.

The 1,000x AI Engineer: Swyx

Oct 23, 2023 · 9:27

Swyx argues that AI engineers are just in time for a 1,000x opportunity, drawing on historical tech cycles and compute scaling laws. He cites Carlota Perez's tech revolution cycles, placing the AI revolution's start at AlexNet in 2012. With compute growing 600x by decade's end, GPT-3 took one person-year of compute, GPT-4 took 100, and GPT-10 will exceed all human compute ever. He defines three AI engineer types: AI Enhanced (Copilot-like), AI Product (Midjourney), and AI Agent (Auto-GPT). To achieve 1,000x, he advises scaling knowledge from O(n) (attending talks) to O(n²) (teaching others) to O(2^n) (building networks).

Announcing the AI Engineer Network: Benjamin Dunphy

Oct 23, 2023 · 7:44

Benjamin Dunphy launches the AI Engineer Summit 2023 by welcoming attendees and recognizing sponsors Auto-GPT, Supabase, Fixie, Microsoft, and Cloudflare, as well as VC Alessio Finelli. He then unveils two major updates for the conference's mobile app, Network: AI-Enhanced Matching, a generative algorithm using LLMs to connect attendees based on stated problems, and the open-sourcing of both the event app and matching algorithm on GitHub, enabling any event to use them. Dunphy credits lead engineer Simon Sturmer for the mobile app, 'Sweezett's teller' for the matching algorithm, and infrastructure partners Dscope (auth) and Supabase (database and PG vector). The announcements aim to improve high-signal networking among the 500 selected engineers, founders, and sponsors at the summit.