Modular AI Orchestration

The State of Model Routing — NVIDIA, Cognition, OpenRouter

Aug 6, 2026 · 48:17

Cognition's Walden Yan, OpenRouter's Alex Atallah, NVIDIA's Tanay Varshney and Carter Abdallah argue routing should orchestrate frontier and cheaper models, not per-task benchmark picks; Devin Fusion cuts Fable-level intelligence cost by 40%. Yan: task-type routing is fragile because a session shifts from codebase question to feature request to live debugging; Devin keeps a frontier model planning while a cheap sidekick executes. Atallah: OpenRouter's auto router sat unused for two years until OpenClaw heartbeats every ten minutes created an app with two intelligence needs; out-of-distribution, small models thrash: Opus scores three times better at a tenth of Haiku's cost on terminal bench. Varshney cites jagged capabilities for up to 10% higher accuracy; Abdallah adds local/cloud routing.

Claude for Long-Horizon Tasks — Lance Martin, Anthropic

Jul 22, 2026 · 25:19

Lance Martin from Anthropic discusses how Claude's increasing task horizon enables asynchronous agents through decoupled architecture, verifier loops, and self-improving memory systems. He explains the shift from short task horizons (10–20 minutes) to 12+ hours, necessitating decoupling the brain (harness) from hands (sandboxes) for reliability and security. Verifier loops using separate contexts allow models to self-correct, demonstrated on the Parameter Golf benchmark with Opus 4.7. Memory systems inspired by human dreaming correct errors in-band, as shown in a Pokémon example where dreaming prevented repeated failures. Finally, org-level harnesses like Claude Tag provide shared identity and context for multiplayer proactive agents.

Your agent architecture has a half-life of 6 months — Dan Farrelly, CTO, Inngest

Jul 21, 2026 · 19:20

Dan Farrelly, CTO and co-founder of Inngest, argues that agent architectures have a half-life of six months because teams couple execution, context, and compute layers together, causing rapid obsolescence when models, frameworks, or patterns change. He presents a mental model of three discrete layers: execution (the brain, for flow, state, durability, retries), context (models, prompts, tools, memory, the layer that changes most), and compute (sandboxes, runtimes, browsers, the hands). Farrelly contends that execution is the stable layer that can last years if properly decoupled, and that it must provide resumability via external durable state, flexible invocation patterns (crons, events, human-in-the-loop), and full session observability beyond LLM calls. He warns against using sandboxes for durability since they are ephemeral and stateless, and instead advises letting execution give sandboxes their context and sequence. Covering emerging trends like background agents and autonomous loops, he emphasizes that these long-running, asynchronous systems require orchestration-aware execution to track failures, debug, and score outcomes. The episode centers on building a harness that…

Make your own event-sourced agent harness using stream processors — Jonas Templestein, Iterate

May 14, 2026 · 1:04:27

Jonas Templestein and Misha from Iterate introduce an event-sourced agent harness built on stream processors, arguing that agents should be modeled as an append-only event log with a synchronous reducer for state and an after-append hook for side effects. They demonstrate how every action—streaming chunks, tool calls, errors—becomes an event, enabling full debuggability and replay without re-running LLM calls. The key innovation is a "dynamic worker configured" event whose payload is a JavaScript processor; appending it to any stream instantly turns that stream into an AI agent with no server or dependencies. This allows processors from different authors and languages to compose on the same stream, and a safety checker can inject context within 200ms without blocking the agent. The hosts emphasize eventual consistency over before-hooks, and show deployment via Cloudflare Workers or by simply subscribing from any HTTP client.

Scaling AI Agents Without Breaking Reliability — Preeti Somal, Temporal

Jul 28, 2025 · 15:01

Preeti Somal, SVP of Engineering at Temporal, argues that the platform is the right choice for building reliable, scalable agentic AI applications. Temporal provides battle-tested orchestration through workflow abstractions, handling state, retries, and failure transparency so developers can focus on business logic. She cites customers like Dust and Gorgias running AI agents on Temporal in production, achieving up to 6x faster feature delivery. The platform uses signals and queries for human interaction, stores full workflow history for debugging and compliance, and offers Temporal Cloud for managed scalability. Somal emphasizes that agents are complex distributed systems requiring robust orchestration, and Temporal’s decade of production use in mission-critical applications proves its reliability.

3 ingredients for building reliable enterprise agents - Harrison Chase, LangChain/LangGraph

Jul 23, 2025 · 20:55

Harrison Chase, CEO of LangChain, argues that reliable enterprise agents depend on three ingredients: high value when right, high probability of success, and low cost when wrong. He says agents succeed by tackling high-value tasks like legal or research, and by doing more work through long-running processes. Reliability improves by blending deterministic workflows with agentic loops, as in LangGraph, and by using observability (LangSmith) to build trust. Cost is lowered by reversible actions (e.g., code diffs) and human-in-the-loop patterns like approvals and time travel. Chase predicts a future of ambient agents—event-triggered, human-supervised background agents—and notes code agents lead because code is verifiable and easily revertible.

Milliseconds to Magic: Real‑Time Workflows using the Gemini Live API and Pipecat

Jun 27, 2025 · 21:43

Kwindla Kramer (Daily, Pipecat) and Shrestha Basu Mallick (Google DeepMind, Gemini API) argue that voice is the most natural interface and that the Gemini Live API combined with the Pipecat framework enables developers to build real-time multimodal voice agents covering the full stack from models to application code. They demo a voice-driven task management app that handles grocery, reading, and work lists, showing impressive context-aware tool use (e.g., consolidating lists, web searching for 'Dream Count' author) but also jagged edge limitations like turn detection errors and persistent misspelling of 'Kwin.' They discuss how capabilities like turn detection migrate down the stack over time. The episode also covers proactivity, multilinguality, telephony integration, and experimental native audio models for emotive, steerable dialogue.

How to build the world's fastest voice bot: Kwindla Hultman Kramer

Feb 10, 2025 · 20:38

Kwindla Hultman Kramer, CEO of Daily, argues that colocating speech-to-text, LLM inference, and text-to-speech in a single compute container is the most effective way to achieve sub-500 millisecond voice-to-voice latency for conversational AI. He details how architectural flexibility and low-latency media transport are critical, citing measured bottlenecks like 30–40ms from macOS mic processing and typical voice-to-voice latencies of 600–700ms. To hit faster response times, his team uses Deepgram’s on-premises STT and TTS models via Docker and Llama 3 8B for LLM inference, achieving 500–700ms in an open-source demo. The talk introduces PipeCat, a vendor-neutral open-source framework for real-time multimodal AI that orchestrates components like transcription, endpointing, interruption handling, and text-to-speech. Kramer emphasizes that while frontier multimodal models are coming, orchestration layers remain essential for building production-grade voice bots, and shares that a recent latency demo gained over 175,000 views on Twitter.

Substrate Launch: the API for modular AI

Feb 6, 2025 · 5:20

Rob Cheung, co-founder of Substrate, presents the platform as a new approach to building AI applications using modular intelligence rather than monolithic models. Substrate combines a developer SDK for describing computation graphs over multiple AI nodes with an inference engine optimized for running these graphs efficiently, reducing latency between nodes to microseconds—10,000 times faster than typical API dispatching. Cheung argues that modular systems are more legible, debuggable, and extensible, enabling higher quality outcomes through explicit decision trees and easier evaluation. The platform offers best-in-class JSON mode for multi-inference runs and supports modalities like image generation, speech transcription, text, embeddings, and code execution. Substrate is designed for developers to run 10-20 AI modules on a single user request at scale, solving the infrastructure problem that currently limits multi-step AI workflows.